mev-beta/tests/calculation-validation

Arbitrage Calculation Validation Framework

This testing framework extracts arbitrage opportunity data from MEV bot logs and validates profit calculations to ensure accuracy and detect bugs.

Overview

The framework consists of three main components:

1. Log Extraction (scripts/test-calculations.sh) - Extracts opportunities, calculations, and threshold checks from logs
2. Validation Library (Go packages) - Parses and validates calculation correctness
3. Replay Tool (replay.go) - Replays calculations and generates validation reports

Quick Start

Extract and Analyze Logs

# Extract from latest container logs
./scripts/test-calculations.sh

# Extract from specific log file
./scripts/test-calculations.sh /path/to/logs.txt

Run Validation Tests

# Run unit tests
go test ./tests/calculation-validation/... -v

# Or in container
podman exec mev-bot-dev-master-dev go test ./tests/calculation-validation/... -v

Replay Calculations

# Replay and validate all calculations
go run ./tests/calculation-validation/replay.go

# With verbose output
go run ./tests/calculation-validation/replay.go -verbose

# Custom test directory
go run ./tests/calculation-validation/replay.go -dir tests/calculation-validation

What It Validates

✅ Profit Threshold Checks

Validates that opportunities are correctly marked as executable based on profit thresholds:

netProfit >= minThreshold  → executable = true
netProfit < minThreshold   → executable = false

Critical Bug Fixed: Before the fix, the code was comparing the integer part of ETH (834) to wei threshold (100000000000000), causing all opportunities to be rejected.

After Fix: Properly converts both values to same units (ETH) for comparison:

834.210302 ETH >= 0.0001 ETH  → EXECUTABLE ✅

✅ ETH to Wei Conversions

Validates that ETH amounts are correctly converted to wei:

1 ETH = 1,000,000,000,000,000,000 wei (1e18)

✅ V3 Swap Calculations

Validates Uniswap V3 swap calculations including:

• Fee deductions (0.05%, 0.3%, 1%)
• Output amounts
• Zero output detection

✅ Opportunity Consistency

Validates logical consistency:

• Executable opportunities have positive profit
• Executable opportunities meet minimum threshold
• Rejected opportunities have valid reject reasons

Output Files

Extraction Phase

tests/calculation-validation/
├── extracted/
│   ├── executable_opportunities.log  # Opportunities marked executable
│   ├── opportunity_details.log       # Full opportunity records
│   ├── v3_calculations.log           # V3 swap calculations
│   ├── threshold_checks.log          # Profit threshold validations
│   ├── rejections.log                # Rejected opportunities
│   ├── profit_values.txt             # Extracted profit amounts
│   └── test_data.json                # Structured test data
└── reports/
    └── validation_report_*.md        # Summary validation report

Example Report

═══════════════════════════════════════════════════════════════════════
  Extraction Complete!
═══════════════════════════════════════════════════════════════════════

Summary Statistics:
  ✅ Executable Opportunities: 11
  📊 Total Opportunity Records: 23
  🔢 V3 Calculations: 600
  ✔️  Threshold Checks: 11
  ❌ Rejections: 23

Profit Statistics:
  💰 Total Profit: 4,053.69 ETH
  📈 Average Profit: 368.517 ETH
  🔝 Max Profit: 1,363.86 ETH
  📉 Min Profit: 0.003708 ETH

Validation Results

After running the replay tool, you'll get:

━━━ Profit Threshold Checks ━━━
  Total: 11
  ✅ Valid: 11
  ❌ Invalid: 0
  Success Rate: 100.00%

━━━ Critical Bug Fix Validation ━━━
  Testing bug fix scenario:
    Net Profit: 834.210302 ETH
    Min Threshold: 0.0001 ETH
    ✅ BUG FIX VALIDATED: Correctly marked as executable

  Testing edge case (profit == threshold):
    Net Profit: 0.0001 ETH
    ✅ Edge case handled correctly

  Testing rejection case (profit < threshold):
    Net Profit: 0.00001 ETH
    ✅ Correctly rejected

Test Coverage

The framework includes comprehensive unit tests:

validator_test.go

• ✅ TestValidateThresholdCheck - Threshold validation logic
• ✅ TestValidateThresholdComparison - Critical bug fix validation
• ✅ TestValidateV3Calculation - Uniswap V3 math
• ✅ TestCompareETHtoWei - Unit conversion
• ✅ TestCalculateStatistics - Profit statistics
• ✅ TestValidateOpportunity - Full opportunity validation

Run tests:

go test ./tests/calculation-validation/... -v -cover

Architecture

Data Flow

┌─────────────────┐
│   Log Files     │
└────────┬────────┘
         │
         ▼
┌─────────────────┐
│  Log Parser     │  (parser.go)
│  - Regex        │
│  - Extraction   │
└────────┬────────┘
         │
         ▼
┌─────────────────┐
│  Test Data      │  (types.go)
│  - Opportunities│
│  - Calculations │
│  - Thresholds   │
└────────┬────────┘
         │
         ▼
┌─────────────────┐
│   Validator     │  (validator.go)
│  - Comparison   │
│  - Validation   │
└────────┬────────┘
         │
         ▼
┌─────────────────┐
│  Test Report    │
│  - Statistics   │
│  - Errors       │
│  - Warnings     │
└─────────────────┘

Use Cases

1. Verify Bug Fixes

After applying a fix to profit calculations, run the framework to validate:

./scripts/test-calculations.sh
go run ./tests/calculation-validation/replay.go -verbose

2. Continuous Integration

Add to CI pipeline to catch calculation regressions:

#!/bin/bash
# Get logs from test run
./mev-bot start &
BOT_PID=$!
sleep 60
kill $BOT_PID

# Extract and validate
./scripts/test-calculations.sh logs/mev_bot.log
go test ./tests/calculation-validation/... || exit 1

3. Performance Analysis

Extract profit statistics over time:

for log in logs/archive/*.log; do
    ./scripts/test-calculations.sh "$log"
done

# Aggregate results
find tests/calculation-validation/reports -name "*.md" -exec cat {} \;

Critical Metrics

The framework tracks key metrics to ensure bot health:

Metric	Healthy Range	Warning	Critical
Executable %	> 5%	1-5%	< 1%
Threshold Pass Rate	100%	95-99%	< 95%
V3 Zero Outputs	< 10%	10-25%	> 25%
Avg Profit	> 0.1 ETH	0.01-0.1 ETH	< 0.01 ETH

Troubleshooting

No Executable Opportunities Found

Symptom: ✓ Found 0 executable opportunities

Possible Causes:

1. Log file is from before the bug fix was applied
2. Bot isn't detecting any profitable opportunities
3. All opportunities are being rejected

Solution:

# Get fresh logs
podman logs mev-bot-dev-master-dev 2>&1 > /tmp/fresh_logs.txt
./scripts/test-calculations.sh /tmp/fresh_logs.txt

# Check for threshold validation
grep "Profit threshold check" /tmp/fresh_logs.txt

Tests Failing

Symptom: go test failures

Solution:

# Check if modules are initialized
cd tests/calculation-validation
go mod init mev-bot/tests/calculation-validation 2>/dev/null || true
go mod tidy

# Run with verbose output
go test -v

Extraction Script Not Found

Symptom: bash: ./scripts/test-calculations.sh: Permission denied

Solution:

chmod +x ./scripts/test-calculations.sh

Future Enhancements

• Real-time validation during bot operation
• Alerting on calculation anomalies
• Historical trend analysis
• Multi-DEX calculation validation
• Gas cost accuracy validation
• Slippage calculation validation

Related Documentation

• Bug Fix Solution - Detailed bug analysis
• Critical Bugs Found - Initial bug discovery
• Profit Calculator - Source code

License

Part of MEV Bot project - see main LICENSE file.