copper-tone-tech/mev-beta

Fork 0

Files

Krypto Kajun 52d555ccdf fix(critical): complete execution pipeline - all blockers fixed and operational

2025-11-04 10:24:34 -06:00

10 KiB

Raw Blame History

MEV Bot Codebase - Quick Reference Guide

Project Snapshot

Language: Go 1.25
Module: github.com/fraktal/mev-beta
Total Files: 362 Go files (253 in pkg/, 31 in internal/, 4 in cmd/)
Total LOC: ~100,000+ (pkg/ only)
Purpose: Production-grade MEV (Maximal Extractable Value) arbitrage bot for Arbitrum

Directory Structure at a Glance

mev-beta/
├── cmd/mev-bot/           → Entry point (786 lines) + DEX integration
├── pkg/                   → 47 packages with core logic
│   ├── arbitrage/         → Arbitrage detection & execution (7000+ LOC)
│   ├── arbitrum/          → Arbitrum L2 integration (8000+ LOC)
│   ├── monitor/           → Sequencer monitoring (1351 LOC)
│   ├── scanner/           → Event processing with worker pools
│   ├── dex/               → Protocol implementations (V3, V2, Curve, Balancer)
│   ├── security/          → Key management & encryption (5000+ LOC)
│   ├── events/            → Event parsing (1806 LOC)
│   ├── transport/         → RPC provider management
│   ├── pools/             → Pool discovery (1065 LOC)
│   ├── math/              → Mathematical operations
│   └── [35 more]          → Specialized packages
├── internal/              → 14 private packages (config, logger, etc.)
├── bindings/              → Go-ethereum generated smart contract ABIs
├── config/                → YAML configuration files
├── tests/                 → Integration & unit tests
└── scripts/               → Build & utility scripts

Core Components (Top 10 by Impact)

#	Component	File	LOC	Role
1	ArbitrageService	pkg/arbitrage/service.go	1,995	Main orchestration
2	L2Parser	pkg/arbitrum/l2_parser.go	1,985	Transaction parsing
3	KeyManager	pkg/security/keymanager.go	1,841	Key & signing
4	EventParser	pkg/events/parser.go	1,806	Event extraction
5	AuditAnalyzer	pkg/security/audit_analyzer.go	1,646	Audit logs
6	ArbitrageExecutor	pkg/arbitrage/executor.go	1,641	Execution
7	FlashSwapExecutor	pkg/arbitrage/flash_executor.go	1,462	Flash swaps
8	RateLimiter	pkg/security/rate_limiter.go	1,411	Rate limiting
9	ArbitrumMonitor	pkg/monitor/concurrent.go	1,351	Monitoring
10	PerformanceProfiler	pkg/security/performance_profiler.go	1,316	Metrics

Data Flow (Simple)

Arbitrum Blocks (WebSocket)
    ↓
ArbitrumMonitor → L2Parser → EventParser
    ↓
Scanner (with Worker Pool)
    ↓
MarketScanner → SwapAnalyzer → LiquidityAnalyzer
    ↓
ArbitrageService → MultiHopScanner → DetectionEngine
    ↓
ArbitrageExecutor → Flash Swap Contract → Arbitrum
    ↓
KeyManager (Signing) → Transaction Submission

Key Architectural Patterns

1. Layered Architecture

Smart Contract Layer (bindings/)
Execution Layer (executor, flash_executor)
Detection Layer (detection_engine, scanner)
Event Collection Layer (monitor, parser)
Infrastructure Layer (transport, pools, security)

2. Worker Pool Pattern

Event processing with configurable workers
Non-blocking channel communication
Graceful shutdown with WaitGroup

3. Rate Limiting

Per-endpoint rate limiters
Global transaction limits
Burst allowances
Automatic backoff

4. Multi-Pool Provider Architecture

Read-only pool (high RPS tolerance)
Execution pool (reliable, limited RPS)
Testing pool (isolated, safe)

5. Configuration Hierarchy

Base: YAML files (config/arbitrum_production.yaml)
Override: Environment variables (GO_ENV, etc.)
Runtime: Per-endpoint config (providers.yaml)

Entry Points & Main Functions

Startup Sequence (main.go - 786 lines)

main()
  ↓ (20 debug checkpoints)
  ├─ Load config (arbitrum_production.yaml)
  ├─ Initialize logger
  ├─ Create provider manager (3-pool architecture)
  ├─ Create key manager (transaction signing)
  ├─ Initialize pool discovery (cache-only)
  ├─ Create market manager
  ├─ Create scanner (worker pools)
  ├─ Create arbitrage service
  ├─ Start monitor loop
  └─ Handle shutdown

Real-Time Loop

ArbitrumMonitor.Start()

for {
    block := subscribe.Next()
    for _, tx := range block.Transactions {
        parsed := l2Parser.ParseTransaction(tx)
        events := eventParser.ParseEvents(receipt)
        scanner.ProcessEvent(events)
    }
}

DEX Protocols Supported

Protocol	File	Status
Uniswap V3	pkg/dex/uniswap_v3.go	Active
Uniswap V2	pkg/dex/	Active
SushiSwap	pkg/dex/sushiswap.go	Active
Curve	pkg/dex/curve.go	Active
Balancer	pkg/dex/balancer.go	Active

Configuration Examples

config/arbitrum_production.yaml

tokens:
  weth: {address: 0x82af..., decimals: 18}
  usdc: {address: 0xaf88..., decimals: 6}
  [20+ tokens defined]

arbitrage:
  min_profit_threshold: 0.1%
  max_slippage: 0.5%
  max_gas_price: 50 gwei

config/providers.yaml

read_only_pool:
  endpoints:
    - url: "https://..."
      max_rps: 50
execution_pool:
  endpoints:
    - url: "https://..."
      max_rps: 20

Environment Variables (Required)

GO_ENV=production
MEV_BOT_ENCRYPTION_KEY=<32-byte hex key>
MEV_BOT_KEYSTORE_PATH=keystore
ARBITRUM_RPC_ENDPOINT=wss://...
ARBITRUM_WS_ENDPOINT=wss://...

Common Build Commands

# Build the binary
make build
cd bin && ./mev-bot start

# Run with specific environment
GO_ENV=production ./bin/mev-bot start

# Run tests
make test                 # All tests
make test-integration     # Integration only
make test-coverage        # With coverage

# Security & Quality
make lint                 # Run linter
make security-scan        # Security audit

Type Definitions (Most Important)

ArbitrageOpportunity

type ArbitrageOpportunity struct {
    ID string
    Path []string              // Token path
    Pools []string             // Pool addresses
    AmountIn *big.Int          // Input amount (wei)
    Profit *big.Int            // Gross profit
    NetProfit *big.Int         // After gas
    GasEstimate *big.Int       // Gas cost
    ROI float64                // Return %
    Confidence float64         // 0-1 score
    TokenIn, TokenOut common.Address
    Timestamp int64
    DetectedAt, ExpiresAt time.Time
}

ArbitrageService

type ArbitrageService struct {
    client *ethclient.Client
    keyManager *security.KeyManager
    executor *ArbitrageExecutor
    detectionEngine *ArbitrageDetectionEngine
    flashExecutor *FlashSwapExecutor
    poolDiscovery *pools.PoolDiscovery
    marketManager *market.MarketManager
}

ArbitrumMonitor

type ArbitrumMonitor struct {
    client *ethclient.Client
    l2Parser *arbitrum.ArbitrumL2Parser
    scanner *scanner.Scanner
    pipeline *market.Pipeline
    eventParser *events.EventParser
}

Execution Flow (Simple)

Opportunity Detected
    ↓
Validate (amount, slippage, confidence)
    ↓
Get execution client
    ↓
Prepare transaction:
    ├─ Select path
    ├─ Estimate gas
    ├─ Calculate slippage
    ├─ Verify profitability
    └─ Sign with KeyManager
    ↓
Submit to Arbitrum
    ↓
Wait for receipt (60s max)
    ↓
Log results & update stats

Known Issues & Workarounds

Issue	Cause	Workaround
Startup hang	Pool discovery RPC calls timeout	Skip discovery, use cache
Security manager off	Debugging hang issue	Commented out in main.go
Multicall parsing	Complex transaction structures	AbiDecoder with fallbacks

Database & Persistence

Current Implementation

In-memory caches for pools and opportunities
Optional PostgreSQL/SQLite for historical data
JSON persistence for pool discovery cache

Key Data

Pool cache: ~314 pools loaded at startup
Opportunity history: In-memory window
Execution history: Configurable retention

Monitoring & Logging

Log Locations

logs/
├── mev_bot.log              # Main application
├── mev_bot_errors.log       # Errors only
├── mev_bot_performance.log  # Metrics
├── analytics/               # Real-time analysis
├── health/                  # Health reports
└── archives/                # Compressed backups

Metrics Available

Health score: 97.97/100
Error rate: 2.03%
MEV opportunities: Count
Processing latency: ~10-50ms
Memory usage: ~200-500MB

Testing

Test Organization

tests/integration/       # Fork testing, real scenarios
tests/cache/            # Cache functionality
tests/contracts/        # Contract interactions
pkg/*_test.go          # Unit tests (colocated)

Running Tests

make test              # All
make test-integration  # Integration only
make test-coverage     # With coverage report
go test -v ./...       # Verbose output

Key Files to Understand First

In Order of Importance:

cmd/mev-bot/main.go - Understand startup flow
pkg/arbitrage/service.go - Core orchestration
pkg/monitor/concurrent.go - How monitoring works
pkg/scanner/concurrent.go - Event processing
pkg/arbitrage/executor.go - Execution logic
pkg/arbitrum/l2_parser.go - Transaction parsing
internal/config/config.go - Configuration system
pkg/security/keymanager.go - Key management

Performance Notes

Throughput: ~100+ events/sec (with 4-8 workers)
Latency: Detection <50ms, execution <5s
Rate Limits: Read 50 RPS, Execute 20 RPS
Memory: ~200-500MB baseline
Disk: Logs rotate, archives compress

Security Notes

Keys encrypted in keystore
All RPC endpoints validated
Rate limiting prevents DoS
Audit logging enabled
Anomaly detection available
Key rotation supported (30-day default)

10 KiB Raw Blame History

MEV Bot Codebase - Quick Reference Guide

Project Snapshot

Directory Structure at a Glance

Core Components (Top 10 by Impact)

Data Flow (Simple)

Key Architectural Patterns

1. Layered Architecture

2. Worker Pool Pattern

3. Rate Limiting

4. Multi-Pool Provider Architecture

5. Configuration Hierarchy

Entry Points & Main Functions

Startup Sequence (main.go - 786 lines)

Real-Time Loop

DEX Protocols Supported

Configuration Examples

config/arbitrum_production.yaml

config/providers.yaml

Environment Variables (Required)

Common Build Commands

Type Definitions (Most Important)

ArbitrageOpportunity

ArbitrageService

ArbitrumMonitor

Execution Flow (Simple)

Known Issues & Workarounds

Database & Persistence

Current Implementation

Key Data

Monitoring & Logging

Log Locations

Metrics Available

Testing

Test Organization

Running Tests

Key Files to Understand First

Performance Notes

Security Notes

See Also

10 KiB

Raw Blame History