mev-beta/docs/spec/MARKET_MANAGER_PLAN.md

Market Manager/BUILDER Planning Document

Overview

This document outlines the plan for implementing a comprehensive Market Manager/BUILDER system for the MEV bot. The system will handle market data collection, storage, and analysis to identify arbitrage opportunities across different DEX protocols on Arbitrum.

Core Requirements

Data Structure

type Market struct {
  Factory     common.Address  // DEX factory contract address
  PoolAddress common.Address  // Pool contract address
  Token0      common.Address  // First token in pair
  Token1      common.Address  // Second token in pair
  Fee         uint32          // Pool fee (e.g., 500 for 0.05%)
  Ticker      string          // Formatted as <symbol>_<symbol> (e.g., "WETH_USDC")
  RawTicker   string          // Formatted as <token0>_<token1> (e.g., "0x..._0x...")
  Key         string          // <keccak256ofToken0Token1FeeFactoryPoolAddress>
}

// Market storage structure
type Markets map[string]map[string]*Market // map[rawTicker]map[marketKey]*Market

Core Functionality

1. Market Data Collection

   • Parse swap and liquidity events from Arbitrum sequencer
   • Store data with "possible" status initially
   • Within 500ms, verify transaction existence on-chain
   • Update data with confirmed on-chain values

2. Market Data Storage

   • Cache market data in memory for fast access
   • Persist data to database for historical analysis
   • Support data versioning (sequencer vs. on-chain)

3. Arbitrage Opportunity Detection

   • Iterate through markets by rawTicker
   • For each rawTicker, examine all associated markets
   • Sort by price (least to highest)
   • Check each combination for arbitrage opportunities
   • Validate profit exceeds threshold (fee1 + fee0 + minArbPct)

Implementation Phases

Phase 1: Market Data Structure and Storage (Week 1)

1.1 Core Data Structures

• Implement Market struct with all required fields
• Implement Markets type (map[rawTicker]map[marketKey]*Market)
• Add helper functions for key generation (keccak256 hashing)
• Implement serialization/deserialization for database storage

1.2 Market Manager Core

• Create MarketManager interface
• Implement in-memory market storage
• Add market CRUD operations (Create, Read, Update, Delete)
• Implement market lookup by various keys (ticker, rawTicker, key)

1.3 Database Integration

• Design database schema for market data persistence
• Implement database adapter for market storage
• Add data versioning support (sequencer vs. on-chain)
• Implement batch operations for efficient data handling

Phase 2: Data Collection and Verification (Week 2)

2.1 Event Parsing Enhancement

• Extend event parser to handle market-specific data
• Implement swap event parsing with full liquidity data
• Add liquidity event parsing (add/remove liquidity)
• Implement new pool event parsing

2.2 Sequencer Data Processing

• Implement sequencer data collection pipeline
• Add "possible" status marking for new market data
• Implement timestamp tracking for verification scheduling
• Add data validation before initial storage

2.3 On-chain Verification

• Implement verification scheduler (500ms window)
• Add Ethereum client integration for transaction verification
• Implement on-chain data retrieval and comparison
• Update market data with confirmed on-chain values

Phase 3: Arbitrage Detection Engine (Week 3)

3.1 Market Iteration and Sorting

• Implement market iteration by rawTicker
• Add price sorting functionality (least to highest)
• Implement efficient market combination generation
• Add performance optimization for large market sets

3.2 Profit Calculation

• Implement fee calculation for different pool types
• Add price impact modeling for large trades
• Implement profit threshold validation
• Add gas cost estimation for arbitrage transactions

3.3 Arbitrage Validation

• Implement arbitrage opportunity detection algorithm
• Add multi-hop arbitrage support
• Implement risk assessment for each opportunity
• Add opportunity scoring and ranking

Phase 4: Performance Optimization and Testing (Week 4)

4.1 Caching and Performance

• Implement intelligent caching strategies
• Add cache warming for frequently accessed markets
• Implement cache expiration and cleanup
• Optimize memory usage for large market datasets

4.2 Testing and Validation

• Implement unit tests for all core functionality
• Add integration tests with mock blockchain data
• Implement performance benchmarks
• Add stress testing for high-volume scenarios

4.3 Monitoring and Observability

• Add metrics collection for market operations
• Implement logging for key events and errors
• Add health checks for market data freshness
• Implement alerting for critical system issues

Technical Considerations

Data Consistency

• Handle race conditions between sequencer data and on-chain verification
• Implement transactional updates for market data
• Add conflict resolution for concurrent data modifications

Scalability

• Design for horizontal scaling across multiple market segments
• Implement sharding for large market datasets
• Add load balancing for data processing tasks

Security

• Validate all incoming market data
• Implement rate limiting for data collection
• Add authentication for market data access
• Implement audit logging for all market operations

Dependencies

1. Existing event parsing infrastructure
2. Ethereum client libraries for on-chain verification
3. Database system for persistence
4. Cache system for in-memory storage

Success Metrics

• Market data processing latency < 100ms
• On-chain verification success rate > 99%
• Arbitrage detection accuracy > 95%
• System uptime > 99.9%
• Memory usage < 2GB for 10,000 markets

Risk Mitigation

1. Data Inconsistency: Implement robust conflict resolution
2. Performance Issues: Add caching and optimize algorithms
3. Network Failures: Implement retry mechanisms with exponential backoff
4. Security Breaches: Add comprehensive input validation and authentication

Timeline

• Week 1: Market Data Structure and Storage
• Week 2: Data Collection and Verification
• Week 3: Arbitrage Detection Engine
• Week 4: Performance Optimization and Testing