# Market Manager/BUILDER Planning Document

## Overview

This document outlines the plan for implementing a comprehensive Market Manager/BUILDER system for the MEV bot. The system will handle market data collection, storage, and analysis to identify arbitrage opportunities across different DEX protocols on Arbitrum.

## Core Requirements

### Data Structure
```go
type Market struct {
  Factory     common.Address  // DEX factory contract address
  PoolAddress common.Address  // Pool contract address
  Token0      common.Address  // First token in pair
  Token1      common.Address  // Second token in pair
  Fee         uint32          // Pool fee (e.g., 500 for 0.05%)
  Ticker      string          // Formatted as <symbol>_<symbol> (e.g., "WETH_USDC")
  RawTicker   string          // Formatted as <token0>_<token1> (e.g., "0x..._0x...")
  Key         string          // <keccak256ofToken0Token1FeeFactoryPoolAddress>
}

// Market storage structure
type Markets map[string]map[string]*Market // map[rawTicker]map[marketKey]*Market
```

### Core Functionality

1. **Market Data Collection**
   - Parse swap and liquidity events from Arbitrum sequencer
   - Store data with "possible" status initially
   - Within 500ms, verify transaction existence on-chain
   - Update data with confirmed on-chain values

2. **Market Data Storage**
   - Cache market data in memory for fast access
   - Persist data to database for historical analysis
   - Support data versioning (sequencer vs. on-chain)

3. **Arbitrage Opportunity Detection**
   - Iterate through markets by rawTicker
   - For each rawTicker, examine all associated markets
   - Sort by price (least to highest)
   - Check each combination for arbitrage opportunities
   - Validate profit exceeds threshold (fee1 + fee0 + minArbPct)

## Implementation Phases

### Phase 1: Market Data Structure and Storage (Week 1)

#### 1.1 Core Data Structures
- [ ] Implement Market struct with all required fields
- [ ] Implement Markets type (map[rawTicker]map[marketKey]*Market)
- [ ] Add helper functions for key generation (keccak256 hashing)
- [ ] Implement serialization/deserialization for database storage

#### 1.2 Market Manager Core
- [ ] Create MarketManager interface
- [ ] Implement in-memory market storage
- [ ] Add market CRUD operations (Create, Read, Update, Delete)
- [ ] Implement market lookup by various keys (ticker, rawTicker, key)

#### 1.3 Database Integration
- [ ] Design database schema for market data persistence
- [ ] Implement database adapter for market storage
- [ ] Add data versioning support (sequencer vs. on-chain)
- [ ] Implement batch operations for efficient data handling

### Phase 2: Data Collection and Verification (Week 2)

#### 2.1 Event Parsing Enhancement
- [ ] Extend event parser to handle market-specific data
- [ ] Implement swap event parsing with full liquidity data
- [ ] Add liquidity event parsing (add/remove liquidity)
- [ ] Implement new pool event parsing

#### 2.2 Sequencer Data Processing
- [ ] Implement sequencer data collection pipeline
- [ ] Add "possible" status marking for new market data
- [ ] Implement timestamp tracking for verification scheduling
- [ ] Add data validation before initial storage

#### 2.3 On-chain Verification
- [ ] Implement verification scheduler (500ms window)
- [ ] Add Ethereum client integration for transaction verification
- [ ] Implement on-chain data retrieval and comparison
- [ ] Update market data with confirmed on-chain values

### Phase 3: Arbitrage Detection Engine (Week 3)

#### 3.1 Market Iteration and Sorting
- [ ] Implement market iteration by rawTicker
- [ ] Add price sorting functionality (least to highest)
- [ ] Implement efficient market combination generation
- [ ] Add performance optimization for large market sets

#### 3.2 Profit Calculation
- [ ] Implement fee calculation for different pool types
- [ ] Add price impact modeling for large trades
- [ ] Implement profit threshold validation
- [ ] Add gas cost estimation for arbitrage transactions

#### 3.3 Arbitrage Validation
- [ ] Implement arbitrage opportunity detection algorithm
- [ ] Add multi-hop arbitrage support
- [ ] Implement risk assessment for each opportunity
- [ ] Add opportunity scoring and ranking

### Phase 4: Performance Optimization and Testing (Week 4)

#### 4.1 Caching and Performance
- [ ] Implement intelligent caching strategies
- [ ] Add cache warming for frequently accessed markets
- [ ] Implement cache expiration and cleanup
- [ ] Optimize memory usage for large market datasets

#### 4.2 Testing and Validation
- [ ] Implement unit tests for all core functionality
- [ ] Add integration tests with mock blockchain data
- [ ] Implement performance benchmarks
- [ ] Add stress testing for high-volume scenarios

#### 4.3 Monitoring and Observability
- [ ] Add metrics collection for market operations
- [ ] Implement logging for key events and errors
- [ ] Add health checks for market data freshness
- [ ] Implement alerting for critical system issues

## Technical Considerations

### Data Consistency
- Handle race conditions between sequencer data and on-chain verification
- Implement transactional updates for market data
- Add conflict resolution for concurrent data modifications

### Scalability
- Design for horizontal scaling across multiple market segments
- Implement sharding for large market datasets
- Add load balancing for data processing tasks

### Security
- Validate all incoming market data
- Implement rate limiting for data collection
- Add authentication for market data access
- Implement audit logging for all market operations

## Dependencies

1. Existing event parsing infrastructure
2. Ethereum client libraries for on-chain verification
3. Database system for persistence
4. Cache system for in-memory storage

## Success Metrics

- Market data processing latency < 100ms
- On-chain verification success rate > 99%
- Arbitrage detection accuracy > 95%
- System uptime > 99.9%
- Memory usage < 2GB for 10,000 markets

## Risk Mitigation

1. **Data Inconsistency**: Implement robust conflict resolution
2. **Performance Issues**: Add caching and optimize algorithms
3. **Network Failures**: Implement retry mechanisms with exponential backoff
4. **Security Breaches**: Add comprehensive input validation and authentication

## Timeline

- Week 1: Market Data Structure and Storage
- Week 2: Data Collection and Verification
- Week 3: Arbitrage Detection Engine
- Week 4: Performance Optimization and Testing