feat: Implement comprehensive Market Manager with database and logging
- Add complete Market Manager package with in-memory storage and CRUD operations - Implement arbitrage detection with profit calculations and thresholds - Add database adapter with PostgreSQL schema for persistence - Create comprehensive logging system with specialized log files - Add detailed documentation and implementation plans - Include example application and comprehensive test suite - Update Makefile with market manager build targets - Add check-implementations command for verification
This commit is contained in:
Krypto Kajun
256
docs/ARBITRAGE_PROFIT_SYSTEM.md
Normal file
256
docs/ARBITRAGE_PROFIT_SYSTEM.md
Normal file
@@ -0,0 +1,256 @@
|
||||
# Enhanced Arbitrage Profit Calculation System
|
||||
|
||||
## Overview
|
||||
|
||||
The MEV bot now includes a sophisticated arbitrage profit calculation and opportunity ranking system that provides real-time analysis of swap events for potential arbitrage opportunities. This system replaces the previous placeholder calculations with comprehensive profit analysis.
|
||||
|
||||
## Key Components
|
||||
|
||||
### 1. SimpleProfitCalculator (`pkg/profitcalc/simple_profit_calc.go`)
|
||||
|
||||
The core profit calculation engine that analyzes swap opportunities:
|
||||
|
||||
#### Features:
|
||||
- **Real-time Gas Price Updates**: Automatically fetches current gas prices from the network
|
||||
- **MEV Competition Modeling**: Adds 50% priority fee boost for competitive MEV transactions
|
||||
- **Comprehensive Profit Analysis**: Calculates gross profit, gas costs, and net profit
|
||||
- **Risk Assessment**: Evaluates confidence scores based on trade characteristics
|
||||
- **Thread-Safe Operations**: Concurrent access to gas price data
|
||||
|
||||
#### Key Methods:
|
||||
```go
|
||||
// Analyze a swap opportunity for arbitrage potential
|
||||
func (spc *SimpleProfitCalculator) AnalyzeSwapOpportunity(
|
||||
ctx context.Context,
|
||||
tokenA, tokenB common.Address,
|
||||
amountIn, amountOut *big.Float,
|
||||
protocol string,
|
||||
) *SimpleOpportunity
|
||||
|
||||
// Update gas price from network (automatic via background goroutine)
|
||||
func (spc *SimpleProfitCalculator) UpdateGasPrice(gasPrice *big.Int)
|
||||
```
|
||||
|
||||
#### Configuration:
|
||||
- **Minimum Profit Threshold**: 0.01 ETH (configurable)
|
||||
- **Gas Limit**: 200,000 gas for simple arbitrage
|
||||
- **Gas Price Updates**: Every 30 seconds from network
|
||||
- **MEV Priority Fee**: 50% boost above base gas price
|
||||
|
||||
### 2. OpportunityRanker (`pkg/profitcalc/opportunity_ranker.go`)
|
||||
|
||||
Advanced filtering and ranking system for arbitrage opportunities:
|
||||
|
||||
#### Features:
|
||||
- **Multi-Factor Scoring**: Composite scores based on profit margin, confidence, trade size, etc.
|
||||
- **Opportunity Deduplication**: Merges similar opportunities and tracks update counts
|
||||
- **Staleness Management**: Automatically removes opportunities older than 5 minutes
|
||||
- **Competition Risk Assessment**: Estimates MEV competition based on profitability
|
||||
- **Configurable Filtering**: Minimum confidence and profit margin thresholds
|
||||
|
||||
#### Ranking Weights:
|
||||
```go
|
||||
DefaultRankingWeights = RankingWeights{
|
||||
ProfitMargin: 0.3, // 30% - profit margin is very important
|
||||
NetProfit: 0.25, // 25% - absolute profit matters
|
||||
Confidence: 0.2, // 20% - confidence in the opportunity
|
||||
TradeSize: 0.1, // 10% - larger trades are preferred
|
||||
Freshness: 0.1, // 10% - fresher opportunities are better
|
||||
Competition: 0.05, // 5% - competition risk (negative)
|
||||
GasEfficiency: 0.1, // 10% - gas efficiency
|
||||
}
|
||||
```
|
||||
|
||||
#### Key Methods:
|
||||
```go
|
||||
// Add and rank a new opportunity
|
||||
func (or *OpportunityRanker) AddOpportunity(opp *SimpleOpportunity) *RankedOpportunity
|
||||
|
||||
// Get top N opportunities by score
|
||||
func (or *OpportunityRanker) GetTopOpportunities(limit int) []*RankedOpportunity
|
||||
|
||||
// Get only executable opportunities
|
||||
func (or *OpportunityRanker) GetExecutableOpportunities(limit int) []*RankedOpportunity
|
||||
```
|
||||
|
||||
### 3. Scanner Integration (`pkg/scanner/concurrent.go`)
|
||||
|
||||
The market scanner now includes integrated profit calculation:
|
||||
|
||||
#### Enhanced logSwapOpportunity Method:
|
||||
- Analyzes each swap event for arbitrage potential
|
||||
- Adds opportunities to the ranking system
|
||||
- Logs comprehensive profit metrics
|
||||
- Includes ranking scores and competition risk in additional data
|
||||
|
||||
#### New Scanner Methods:
|
||||
```go
|
||||
// Access top ranked opportunities
|
||||
func (s *MarketScanner) GetTopOpportunities(limit int) []*profitcalc.RankedOpportunity
|
||||
|
||||
// Get executable opportunities ready for trading
|
||||
func (s *MarketScanner) GetExecutableOpportunities(limit int) []*profitcalc.RankedOpportunity
|
||||
|
||||
// Get ranking system statistics
|
||||
func (s *MarketScanner) GetOpportunityStats() map[string]interface{}
|
||||
```
|
||||
|
||||
## Profit Calculation Algorithm
|
||||
|
||||
### 1. Basic Profit Estimation
|
||||
```go
|
||||
// Estimate profit as 0.5% of trade amount (simplified model)
|
||||
grossProfit := new(big.Float).Mul(amountOut, big.NewFloat(0.005))
|
||||
```
|
||||
|
||||
### 2. Gas Cost Calculation
|
||||
```go
|
||||
// Gas cost with MEV competition buffer
|
||||
gasCost = (gasPrice * gasLimit) * 1.2 // 20% buffer
|
||||
```
|
||||
|
||||
### 3. Net Profit Calculation
|
||||
```go
|
||||
netProfit = grossProfit - gasCost
|
||||
```
|
||||
|
||||
### 4. Profit Margin
|
||||
```go
|
||||
profitMargin = netProfit / amountOut
|
||||
```
|
||||
|
||||
### 5. Confidence Scoring
|
||||
Based on multiple factors:
|
||||
- Positive net profit (+40% base confidence)
|
||||
- Profit margin thresholds (>2%, >1%, >0.5%)
|
||||
- Trade size (>$1000, >$100 equivalents)
|
||||
- Capped at 100% confidence
|
||||
|
||||
## Opportunity Ranking System
|
||||
|
||||
### Composite Score Calculation
|
||||
Each opportunity receives a score (0-1) based on weighted factors:
|
||||
|
||||
1. **Profit Margin Score**: Normalized by 10% margin = 1.0
|
||||
2. **Net Profit Score**: Normalized by 0.1 ETH = 1.0
|
||||
3. **Confidence Score**: Direct 0-1 mapping
|
||||
4. **Trade Size Score**: Normalized by $10k = 1.0
|
||||
5. **Freshness Score**: Decays from 1.0 over 5 minutes
|
||||
6. **Competition Risk**: Negative impact based on profitability
|
||||
7. **Gas Efficiency**: Profit per gas ratio
|
||||
|
||||
### Filtering Criteria
|
||||
Opportunities must meet minimum thresholds:
|
||||
- **Confidence**: ≥ 30%
|
||||
- **Profit Margin**: ≥ 0.1%
|
||||
- **Net Profit**: Must be positive
|
||||
|
||||
## Integration with MEV Bot
|
||||
|
||||
### Scanner Enhancement
|
||||
The market scanner now:
|
||||
1. Creates profit calculator with network client for gas price updates
|
||||
2. Creates opportunity ranker for filtering and prioritization
|
||||
3. Analyzes every swap event for arbitrage potential
|
||||
4. Logs detailed profit metrics including ranking scores
|
||||
|
||||
### Logging Enhancement
|
||||
Each opportunity log now includes:
|
||||
```json
|
||||
{
|
||||
"arbitrageId": "arb_1758117537_0x82aF49",
|
||||
"isExecutable": true,
|
||||
"rejectReason": "",
|
||||
"confidence": 0.75,
|
||||
"profitMargin": 0.0125,
|
||||
"netProfitETH": "0.025000",
|
||||
"gasCostETH": "0.000240",
|
||||
"estimatedProfitETH": "0.025240",
|
||||
"opportunityScore": 0.8234,
|
||||
"opportunityRank": 1,
|
||||
"competitionRisk": 0.65,
|
||||
"updateCount": 1
|
||||
}
|
||||
```
|
||||
|
||||
## Performance Characteristics
|
||||
|
||||
### Gas Price Updates
|
||||
- **Frequency**: Every 30 seconds
|
||||
- **MEV Boost**: 50% priority fee above base price
|
||||
- **Thread Safety**: Concurrent access with RWMutex
|
||||
- **Fallback**: 1 gwei default if network fetch fails
|
||||
|
||||
### Opportunity Management
|
||||
- **Capacity**: Tracks up to 50 top opportunities
|
||||
- **TTL**: 5-minute lifetime for opportunities
|
||||
- **Deduplication**: Merges similar token pairs within 10% amount variance
|
||||
- **Cleanup**: Automatic removal of stale opportunities
|
||||
|
||||
### Memory Usage
|
||||
- Minimal memory footprint with automatic cleanup
|
||||
- Efficient sorting and ranking algorithms
|
||||
- No persistent storage (in-memory only)
|
||||
|
||||
## Usage Examples
|
||||
|
||||
### Basic Profit Analysis
|
||||
```go
|
||||
calc := profitcalc.NewSimpleProfitCalculator(logger)
|
||||
opp := calc.AnalyzeSwapOpportunity(ctx, tokenA, tokenB, amountIn, amountOut, "UniswapV3")
|
||||
if opp.IsExecutable {
|
||||
log.Printf("Profitable opportunity: %s ETH profit", calc.FormatEther(opp.NetProfit))
|
||||
}
|
||||
```
|
||||
|
||||
### Opportunity Ranking
|
||||
```go
|
||||
ranker := profitcalc.NewOpportunityRanker(logger)
|
||||
rankedOpp := ranker.AddOpportunity(opp)
|
||||
topOpps := ranker.GetTopOpportunities(5)
|
||||
for _, top := range topOpps {
|
||||
log.Printf("Rank %d: Score %.4f", top.Rank, top.Score)
|
||||
}
|
||||
```
|
||||
|
||||
### Scanner Integration
|
||||
```go
|
||||
scanner := scanner.NewMarketScanner(cfg, logger, executor, db)
|
||||
executable := scanner.GetExecutableOpportunities(3)
|
||||
stats := scanner.GetOpportunityStats()
|
||||
```
|
||||
|
||||
## Future Enhancements
|
||||
|
||||
### Planned Improvements
|
||||
1. **Multi-DEX Price Comparison**: Real-time price feeds from multiple DEXs
|
||||
2. **Slippage Protection**: Advanced slippage modeling for large trades
|
||||
3. **Flash Loan Integration**: Calculate opportunities requiring flash loans
|
||||
4. **Historical Performance**: Track execution success rates and actual profits
|
||||
5. **Machine Learning**: ML-based profit prediction models
|
||||
|
||||
### Configuration Tuning
|
||||
- Adjustable ranking weights based on market conditions
|
||||
- Dynamic gas price multipliers for MEV competition
|
||||
- Configurable opportunity TTL and capacity limits
|
||||
- Protocol-specific profit calculation models
|
||||
|
||||
## Monitoring and Debugging
|
||||
|
||||
### Key Metrics to Monitor
|
||||
- Average opportunity scores
|
||||
- Executable opportunity count
|
||||
- Gas price update frequency
|
||||
- Opportunity staleness rates
|
||||
- Profit calculation accuracy
|
||||
|
||||
### Debug Logging
|
||||
Enable debug logging to see:
|
||||
- Individual opportunity analysis results
|
||||
- Gas price updates from network
|
||||
- Opportunity ranking and scoring details
|
||||
- Filtering decisions and rejection reasons
|
||||
|
||||
## Conclusion
|
||||
|
||||
The enhanced arbitrage profit calculation system provides a solid foundation for MEV opportunity detection and evaluation. The modular design allows for easy extension and customization while maintaining high performance and accuracy in real-time trading scenarios.
|
||||
Reference in New Issue
Block a user