mev-beta/docs/3_core_packages/SCANNER_PACKAGE.md

Scanner Package Documentation

Overview

The scanner package provides comprehensive market scanning capabilities for detecting arbitrage opportunities in decentralized exchanges. It implements concurrent processing, sophisticated profit calculations, market data logging, and advanced arbitrage detection algorithms.

Core Components

MarketScanner Structure

The main market scanner with concurrency support:

1. Configuration - Bot configuration settings
2. Logger - Structured logging
3. WorkerPool - Channel for worker job distribution
4. Workers - Array of event workers
5. WaitGroup - Synchronization for worker completion
6. CacheGroup - Singleflight cache management
7. Cache - In-memory data caching
8. SlippageProtector - Trading slippage protection
9. CircuitBreaker - Fault tolerance mechanism
10. ContractExecutor - Smart contract execution
11. CREATE2Calculator - Pool address calculation
12. Database - Data persistence
13. ProfitCalculator - Profit analysis
14. OpportunityRanker - Opportunity ranking
15. MarketDataLogger - Enhanced market data logging

EventWorker Structure

Worker that processes event details:

1. ID - Worker identifier
2. WorkerPool - Channel for job distribution
3. JobChannel - Channel for receiving jobs
4. QuitChan - Channel for worker termination
5. Scanner - Reference to parent scanner

Initialization

NewMarketScanner(cfg *config.BotConfig, logger *logger.Logger, contractExecutor *contracts.ContractExecutor, db *database.Database) *MarketScanner

Creates a new market scanner with concurrency support:

1. Configuration - Sets up scanner configuration
2. Worker Pool - Initializes worker pool
3. Cache - Sets up caching system
4. Components - Initializes all sub-components
5. Workers - Creates and starts worker pool
6. Cleanup - Starts cache cleanup routine

NewEventWorker(id int, workerPool chan chan events.Event, scanner *MarketScanner) *EventWorker

Creates a new event worker:

1. ID Assignment - Assigns worker ID
2. Channel Setup - Configures communication channels
3. Scanner Link - Links to parent scanner

Concurrent Processing

Worker Management

Start()

Begins the worker processing loop:

1. Registration - Registers worker in pool
2. Job Processing - Processes jobs from channel
3. Termination - Handles quit signals

Stop()

Terminates the worker:

1. Quit Signal - Sends termination signal
2. Cleanup - Performs necessary cleanup

Process(event events.Event)

Handles an event detail:

1. Analysis - Analyzes event in goroutine
2. Routing - Routes to appropriate analyzer
3. Logging - Logs processing information

Event Processing

SubmitEvent(event events.Event)

Submits an event for processing by the worker pool:

1. Worker Allocation - Gets available worker
2. Job Submission - Sends job to worker
3. Synchronization - Manages processing completion

Event Analysis

Swap Event Analysis

analyzeSwapEvent(event events.Event)

Analyzes a swap event for arbitrage opportunities:

1. Pool Data - Retrieves comprehensive pool information
2. Swap Data - Creates detailed swap event data
3. USD Values - Calculates USD values for amounts
4. Price Impact - Calculates price impact
5. Logging - Logs swap event to market data logger
6. Opportunity - Logs swap opportunity with actual amounts
7. Movement - Calculates price movement
8. Significance - Determines movement significance
9. Arbitrage - Searches for arbitrage opportunities
10. Execution - Executes profitable opportunities

logSwapOpportunity(event events.Event, poolData interface{}, priceMovement *PriceMovement)

Logs swap opportunities using actual amounts from events:

1. Amount Conversion - Converts big.Int to big.Float
2. Profit Analysis - Analyzes arbitrage opportunity
3. Ranking - Adds opportunity to ranking system
4. Token Resolution - Resolves token symbols
5. Data Compilation - Compiles additional data
6. Logging - Logs opportunity with profit analysis

Liquidity Event Analysis

analyzeLiquidityEvent(event events.Event, isAdd bool)

Analyzes liquidity events (add/remove):

1. Pool Data - Retrieves comprehensive pool information
2. Liquidity Data - Creates detailed liquidity event data
3. USD Values - Calculates USD values for amounts
4. Logging - Logs liquidity event to market data logger
5. Cache Update - Updates cached pool data
6. Completion - Logs successful processing

New Pool Analysis

analyzeNewPoolEvent(event events.Event)

Analyzes new pool creation events:

1. Validation - Validates new pool contract
2. Data Fetching - Fetches pool data
3. Logging - Logs pool data to database
4. Monitoring - Adds pool to monitoring

Arbitrage Detection

Price Movement Analysis

calculatePriceMovement(event events.Event, poolData *CachedData) (*PriceMovement, error)

Calculates the price movement from a swap event:

1. Current Price - Gets current price from pool data
2. Price Impact - Calculates price impact based on swap amounts
3. Movement Data - Creates price movement structure
4. Logging - Logs calculation details

isSignificantMovement(movement *PriceMovement, threshold float64) bool

Determines if a price movement is significant enough to exploit:

1. Price Impact - Checks price impact threshold
2. Amount Size - Checks absolute amount significance
3. Combined Logic - Uses combined significance logic

Related Pool Discovery

findRelatedPools(token0, token1 common.Address) []*CachedData

Finds pools that trade the same token pair:

1. Pool Discovery - Discovers pools for token pair
2. Validation - Validates pool token pairs
3. Filtering - Filters for matching pairs

discoverPoolsForPair(token0, token1 common.Address) []string

Discovers pools for a specific token pair using real factory contracts:

1. CREATE2 Calculation - Uses CREATE2 calculator
2. Address Conversion - Converts to string addresses
3. Logging - Logs discovery results

Arbitrage Opportunity Detection

findArbitrageOpportunities(event events.Event, movement *PriceMovement) []stypes.ArbitrageOpportunity

Looks for arbitrage opportunities based on price movements:

1. Related Pools - Finds related pools for same token pair
2. Price Comparison - Compares prices between pools
3. Profit Estimation - Estimates potential profits
4. Opportunity Creation - Creates arbitrage opportunities
5. Triangular Search - Searches for triangular opportunities

findTriangularArbitrageOpportunities(event events.Event) []stypes.ArbitrageOpportunity

Looks for triangular arbitrage opportunities:

1. Path Discovery - Gets triangular paths from configuration
2. Token Matching - Matches event tokens to paths
3. Profit Calculation - Calculates triangular profits
4. Opportunity Creation - Creates triangular opportunities

calculateTriangularProfit(tokens []common.Address, initialAmount *big.Int) (*big.Int, *big.Int, error)

Calculates the profit from a triangular arbitrage path:

1. Path Validation - Validates token path length
2. Trading Simulation - Simulates trading through path
3. Pool Discovery - Finds pools for each token pair
4. Output Calculation - Calculates swap outputs
5. Gas Estimation - Estimates gas costs
6. Profit Calculation - Calculates net profit

Profit Calculation

Sophisticated Profit Analysis

estimateProfit(event events.Event, pool *CachedData, priceDiff float64) *big.Int

Estimates the potential profit from an arbitrage opportunity:

1. Slippage Protection - Uses slippage protection if available
2. Fallback Calculation - Uses sophisticated calculation as fallback

calculateProfitWithSlippageProtection(event events.Event, pool *CachedData, priceDiff float64) *big.Int

Uses slippage protection for accurate profit estimation:

1. Trade Parameters - Creates trade parameters from event data
2. Slippage Analysis - Analyzes slippage protection
3. Profit Calculation - Calculates profit considering slippage
4. Gas Costs - Calculates sophisticated gas costs
5. Safety Margin - Applies slippage safety margin

calculateSophisticatedProfit(event events.Event, pool *CachedData, priceDiff float64) *big.Int

Provides advanced profit calculation with MEV considerations:

1. Output Calculation - Calculates swap output using V3 math
2. Market Impact - Calculates market impact
3. Gross Profit - Calculates gross arbitrage profit
4. Cost Estimation - Estimates dynamic gas costs
5. MEV Premium - Calculates MEV competition premium
6. Slippage Tolerance - Applies slippage tolerance
7. Threshold Check - Ensures minimum profit threshold

Mathematical Calculations

calculateUniswapV3Output(amountIn *big.Int, pool *CachedData) (*big.Int, error)

Calculates swap output using proper Uniswap V3 concentrated liquidity math:

1. Formula Application - Applies concentrated liquidity formula
2. New Price Calculation - Calculates new sqrt price
3. Output Amount - Calculates output amount
4. Fee Application - Applies pool fee
5. Final Output - Returns final output amount

calculateMarketImpact(amountIn *big.Int, pool *CachedData) *big.Int

Estimates the market impact of a large trade:

1. Impact Ratio - Calculates amount to liquidity ratio
2. Quadratic Impact - Applies quadratic impact model
3. Capping - Caps maximum impact

calculateDynamicGasCost(event events.Event, pool *CachedData) *big.Int

Calculates gas cost based on current network conditions:

1. Base Gas - Determines base gas for operation
2. Gas Price - Gets current gas price
3. Priority Fee - Adds MEV priority fee
4. Total Cost - Calculates total gas cost

calculateMEVPremium(grossProfit *big.Int, priceDiff float64) *big.Int

Calculates the premium needed to compete with other MEV bots:

1. Base Premium - Calculates base 5% premium
2. Competition Scaling - Scales with price difference
3. Capping - Caps at 30% of gross profit

calculateSlippageTolerance(amountIn *big.Int, pool *CachedData) *big.Int

Calculates acceptable slippage for the trade:

1. Base Tolerance - Calculates 0.5% base tolerance
2. Liquidity Scaling - Scales with trade to liquidity ratio
3. Capping - Caps at 2% maximum

Arbitrage Execution

Opportunity Execution

executeArbitrageOpportunity(opportunity stypes.ArbitrageOpportunity)

Executes an arbitrage opportunity using the smart contract:

1. Profit Validation - Checks minimum profit threshold
2. Contract Availability - Validates contract executor
3. Opportunity Type - Determines execution method
4. Transaction Submission - Submits arbitrage transaction
5. Logging - Logs execution results

Data Management

Caching System

getPoolData(poolAddress string) (*CachedData, error)

Retrieves pool data with caching:

1. Cache Check - Checks cache for existing data
2. Freshness Validation - Validates cache expiration
3. Singleflight - Prevents duplicate requests
4. Data Fetching - Fetches data if needed
5. Cache Update - Updates cache with new data

fetchPoolData(poolAddress string) (*CachedData, error)

Fetches pool data from the blockchain:

1. Test Environment - Returns mock data if testing
2. Client Connection - Connects to Ethereum node
3. Pool Validation - Validates pool contract
4. State Fetching - Fetches real pool state
5. Data Creation - Creates pool data structure

updatePoolData(event events.Event)

Updates cached pool data from an event:

1. Cache Locking - Acquires cache mutex
2. Data Update - Updates existing cache entry
3. New Entry - Creates new entry if needed
4. Database Logging - Logs updated data
5. Cache Unlocking - Releases cache mutex

cleanupCache()

Removes expired cache entries:

1. Ticker Setup - Sets up periodic cleanup
2. Entry Validation - Checks cache entry expiration
3. Entry Removal - Removes expired entries
4. Logging - Logs cleanup activity

Database Integration

logSwapEvent(event events.Event)

Logs a swap event to the database:

1. Data Conversion - Converts event to database record
2. Asynchronous Logging - Logs asynchronously to avoid blocking
3. Error Handling - Handles logging errors

logLiquidityEvent(event events.Event, eventType string)

Logs a liquidity event to the database:

1. Data Conversion - Converts event to database record
2. Asynchronous Logging - Logs asynchronously to avoid blocking
3. Error Handling - Handles logging errors

logPoolData(poolData *CachedData)

Logs pool data to the database:

1. Data Conversion - Converts cached data to database record
2. Asynchronous Logging - Logs asynchronously to avoid blocking
3. Error Handling - Handles logging errors

Helper Functions

Token Resolution

resolveTokenSymbol(tokenAddress string) string

Converts token address to human-readable symbol:

1. Address Normalization - Converts to lowercase
2. Mapping Lookup - Looks up in known token mappings
3. Fallback - Returns truncated address if unknown

Protocol Detection

getFactoryForProtocol(protocol string) common.Address

Returns the factory address for a known DEX protocol:

1. Known Factories - Maps protocols to factory addresses
2. Default Handling - Returns default for unknown protocols

Environment Detection

isTestEnvironment() bool

Checks if we're running in a test environment:

1. Environment Variable - Checks GO_TEST variable
2. Argument Analysis - Analyzes command line arguments
3. Program Name - Checks program name patterns

Mock Data

getMockPoolData(poolAddress string) *CachedData

Returns mock pool data for testing:

1. Token Selection - Selects tokens based on pool address
2. Data Generation - Generates deterministic mock data
3. Structure Creation - Creates cached data structure

Data Structures

PriceMovement Structure

Represents a potential price movement:

• Token0/Token1 - Token addresses
• Pool - Pool address
• Protocol - DEX protocol
• AmountIn/AmountOut - Swap amounts
• PriceBefore/PriceAfter - Price before/after swap
• PriceImpact - Calculated price impact
• TickBefore/TickAfter - Tick before/after swap
• Timestamp - Event timestamp

CachedData Structure

Represents cached pool data:

• Address - Pool contract address
• Token0/Token1 - Pool token addresses
• Fee - Pool fee tier
• Liquidity - Current liquidity (uint256)
• SqrtPriceX96 - Square root price (uint256)
• Tick - Current tick
• TickSpacing - Tick spacing for fee tier
• Protocol - DEX protocol
• LastUpdated - Last update timestamp

Performance Optimization

Concurrency Management

The scanner implements efficient concurrent processing:

1. Worker Pool - Fixed-size worker pool
2. Job Distribution - Efficient job distribution
3. Channel Management - Buffered channel communication
4. Synchronization - Proper wait group usage
5. Resource Cleanup - Graceful worker termination

Caching Strategy

Intelligent caching with multiple optimization techniques:

1. Singleflight - Prevents duplicate requests
2. Time-based Expiration - Automatic cache expiration
3. Memory Management - Efficient data structures
4. Cleanup Routines - Periodic cache cleanup
5. Test Environment - Mock data for testing

Mathematical Efficiency

Optimized mathematical calculations:

1. Precached Constants - Cached mathematical constants
2. BigInt Operations - Efficient big integer operations
3. Floating Point - Appropriate float64 usage
4. Formula Optimization - Simplified mathematical formulas
5. Error Handling - Graceful mathematical error handling

Best Practices

Concurrent Design

1. Worker Pool Sizing - Match workers to hardware capabilities
2. Channel Buffering - Use appropriate buffer sizes
3. Context Management - Proper context cancellation
4. Resource Cleanup - Ensure proper resource release
5. Error Propagation - Clear error communication

Data Management

1. Cache Strategy - Implement appropriate TTL
2. Memory Efficiency - Minimize memory allocations
3. Database Integration - Asynchronous database operations
4. Data Validation - Validate all incoming data
5. Fallback Mechanisms - Provide fallback data sources

Mathematical Accuracy

1. Precision Handling - Use appropriate data types
2. Error Bounds - Understand calculation error bounds
3. Validation - Validate mathematical inputs
4. Testing - Comprehensive mathematical testing
5. Documentation - Document mathematical formulas

Future Improvements

Enhanced Features

1. Machine Learning - ML-based opportunity prediction
2. Advanced Analytics - Sophisticated market analysis
3. Cross-Chain - Multi-chain arbitrage detection
4. Real-time Updates - WebSocket-based market updates
5. Advanced Caching - Distributed caching systems

Performance Enhancements

1. Algorithmic Improvements - Faster mathematical calculations
2. Memory Optimization - Reduced memory allocations
3. Network Efficiency - Optimized RPC usage
4. Batch Processing - Enhanced batch operations
5. Compression - Data compression for caching

Security Improvements

1. Enhanced Validation - More thorough data validation
2. Rate Limiting - Advanced rate limiting strategies
3. Access Control - Enhanced access controls
4. Audit Logging - Comprehensive audit trails
5. Threat Detection - Advanced threat detection