mev-beta/docs/PRODUCTION_REMEDIATION_ACTION_PLAN_20251106.md

MEV Bot Production Remediation - Comprehensive Action Plan

Date: November 6, 2025 Status: IN EXECUTION - Critical Issues Identified and Fixing Priority: CRITICAL - Multiple blockers to production deployment

---

EXECUTIVE SUMMARY

Current State:

• ✅ Format string compile error FIXED
• ⚠️ Tests EXIST (71 files) and RUN but have FAILURES
• ❌ CRITICAL packages missing tests (profitcalc, exchanges, tokens, etc.)
• ❌ Test coverage and profitability validation PENDING
• ⏳ Full bot execution and validation NOT YET DONE

Decision: DO NOT DEPLOY UNTIL:

1. All failing tests fixed ✅
2. Missing tests created for critical packages ✅
3. Code coverage ≥ 80% minimum ✅
4. Bot execution validates opportunity detection ✅

---

CRITICAL FINDINGS

1. FORMAT STRING ERROR ✅ RESOLVED

• File: pkg/profitcalc/profit_calc.go:277
• Issue: (> 1000%) should be (> 1000%%)
• Status: FIXED - Build now succeeds
• Action: COMPLETED

2. TEST FAILURES DETECTED 🔴 REQUIRES IMMEDIATE FIX

Failing Tests in pkg/arbitrage:

FAIL: TestNewMultiHopScanner
  - Expected 4 paths, got 3
  - Expected amount 1000000000000000, got 10000000000000
  - Expected 0.03 fee, got 0.05
  - Expected 100 confidence, got 200
  - Expected 500ms timeout, got 2s

FAIL: TestEstimateHopGasCost
  - Expected 150000 gas for hop 1, got 70000
  - Expected 120000 gas for hop 2, got 60000
  - Expected 120000 gas for hop 3, got 60000

Action: Must fix these test assertions or correct the implementation

3. MISSING CRITICAL TEST FILES 🔴 HIGH PRIORITY

Packages WITH code but NO tests:

• pkg/profitcalc (CRITICAL - profit calculations!)
• pkg/exchanges (DEX interactions)
• pkg/tokens (token handling)
• pkg/execution (trade execution)
• pkg/trading
• pkg/oracle
• pkg/performance
• pkg/patterns
• pkg/dex
• And 10+ more packages

Action: Must create tests for all critical packages

4. ZERO CODE COVERAGE ISSUE 🟡 INVESTIGATION COMPLETE

• Issue: Earlier runs showed 0.0% coverage despite tests existing
• Root Cause: Output buffering and tee issues (not a real problem)
• Resolution: Actual tests ARE running and show coverage
• Status: In progress - full test run underway

---

IMMEDIATE ACTION PLAN (NEXT 24 HOURS)

Phase 1: Analyze Full Test Results (NOW - 30 min)

When full test run completes:

# Check total test status
go tool cover -func=coverage-full.out | tail -5

# List all failing tests
grep "FAIL:" full-test-results.log | sort | uniq

# Get coverage summary
grep "coverage:" full-test-results.log | sort | uniq -c

Expected Outcome: Clear list of failures and coverage percentage

Phase 2: Fix Failing Tests (1-2 hours)

For TestNewMultiHopScanner failures:

1. Review test assertions in pkg/arbitrage/multihop_test.go:60-64
2. Verify if test expectations are wrong OR implementation is wrong
3. Either fix test or fix implementation
4. Re-run tests to verify pass

For TestEstimateHopGasCost failures:

1. Review gas estimation logic in pkg/arbitrage/multihop.go
2. Check if hardcoded gas values match actual costs
3. Fix either test or implementation
4. Re-run and verify

Phase 3: Create Missing Tests for Critical Packages (4-8 hours)

Priority 1 (MUST HAVE):

• pkg/profitcalc/*_test.go - Profit calculation tests
• pkg/execution/*_test.go - Trade execution tests
• pkg/exchanges/*_test.go - DEX interaction tests

Priority 2 (SHOULD HAVE):

• pkg/tokens/*_test.go - Token handling tests
• pkg/trading/*_test.go - Trading logic tests
• pkg/oracle/*_test.go - Price oracle tests

Priority 3 (NICE TO HAVE):

• pkg/dex/*_test.go - DEX adapter tests
• pkg/performance/*_test.go - Performance tracking tests
• pkg/patterns/*_test.go - Pattern matching tests

Phase 4: Verify Test Coverage (30 min)

# Generate coverage report
go test -v -coverprofile=coverage-final.out ./pkg/... ./internal/...
go tool cover -func=coverage-final.out | tail -1

# Target: ≥ 80% coverage
# Current: TBD (waiting for full test results)

Phase 5: Validate Profitability Configuration (1 hour)

Review and validate:

// File: pkg/profitcalc/profit_calc.go
minProfitThreshold = 0.001 ETH  // ← May be too high!
maxSlippage = 3% (0.03)
gasLimit = 100,000
gasPrice = 0.1 gwei + dynamic

Actions:

1. Check if 0.001 ETH threshold is realistic for Arbitrum
2. Verify gas estimation is accurate
3. Test with mock market data to validate profitability detection

Phase 6: Run Bot and Validate Execution (1-2 hours)

# Build release binary
make build

# Run with full logging
LOG_LEVEL=debug METRICS_ENABLED=true timeout 300 ./bin/mev-bot start

# Check logs for:
# - Opportunity detections (should see > 0)
# - Successful executions
# - Error rates (should be low)
# - Performance metrics

---

DETAILED FIX CHECKLIST

Section A: TEST FAILURES

A1. Fix TestNewMultiHopScanner

Location: pkg/arbitrage/multihop_test.go:60-64

❌ FAIL: expected 4, actual 3
❌ FAIL: expected "1000000000000000", actual "10000000000000"
❌ FAIL: expected 0.03, actual 0.05
❌ FAIL: expected 100, actual 200
❌ FAIL: expected 500ms, actual 2s

Investigation needed:

1. Is test data outdated?
2. Did implementation change?
3. Is there a legitimate calculation difference?

Options:

• Update test expectations if implementation is correct
• Fix implementation if test expectations are correct
• Review git history to understand change

A2. Fix TestEstimateHopGasCost

Location: pkg/arbitrage/multihop_test.go:252-264

❌ FAIL: expected 150000, actual 70000 (hop 1)
❌ FAIL: expected 120000, actual 60000 (hop 2)
❌ FAIL: expected 120000, actual 60000 (hop 3)

Investigation needed:

1. Are gas estimations too low?
2. Are test expectations outdated from earlier audits?
3. Is Arbitrum gas model different from expected?

Actions:

• Verify Arbitrum L2 gas prices vs assumptions
• Check if gas can be estimated more accurately
• Update test or implementation

Section B: MISSING TESTS

B1. Create profitcalc_test.go (CRITICAL)

// pkg/profitcalc/profitcalc_test.go
// Test coverage needed for:
// - ProfitCalculator initialization
// - CalculateOpportunity function
// - Profit margin calculations
// - Slippage validation
// - Gas cost estimation
// - Confidence scoring

B2. Create exchanges_test.go (CRITICAL)

// pkg/exchanges/exchanges_test.go
// Test coverage needed for:
// - DEX adapter initialization
// - Price fetch operations
// - Liquidity pool interactions
// - Fee calculations

B3. Create execution_test.go (CRITICAL)

// pkg/execution/execution_test.go
// Test coverage needed for:
// - Transaction building
// - Execution strategy selection
// - Flash loan integration
// - Success/failure handling

B4. Create tokens_test.go (HIGH)

// pkg/tokens/tokens_test.go
// Test coverage needed for:
// - Token metadata caching
// - Decimal handling
// - Symbol/address resolution

Section C: PROFITABILITY VALIDATION

C1. Verify Min Profit Threshold

Current: 0.001 ETH = $2.00 at $2000/ETH Question: Is this realistic for MEV opportunities?

Steps:

1. Research typical Arbitrum arbitrage spreads
2. Check if threshold filters out viable trades
3. Consider lowering to 0.0001 ETH if needed

C2. Verify Gas Estimation

Current: Hardcoded 100k gas limit Question: Accurate for all transaction types?

Steps:

1. Test with real Arbitrum transactions
2. Verify actual gas costs vs estimated
3. Implement adaptive gas estimation if needed

C3. Validate against market data

1. Test profit calculation with real price feeds
2. Verify slippage protection
3. Check flash loan handling

Section D: MAKEFILE OPTIMIZATION FOR PODMAN

D1. Audit Makefile targets

# Check which commands use Docker vs Podman
grep -r "docker\|Docker" Makefile
grep -r "podman\|Podman" Makefile

# Expected: All commands should be Podman-first

D2. Update commands

• Build targets - use Podman
• Test targets - use Podman Compose
• CI targets - use Podman
• Deploy targets - use Podman

---

TIMELINE & DEPENDENCIES

Phase | Task | Duration | Depends On | Status
------|------|----------|-----------|--------
1 | Analyze full tests | 30 min | Tests complete | ⏳ WAITING
2 | Fix test failures | 1-2 hrs | Phase 1 | ⏳ WAITING
3 | Create missing tests | 4-8 hrs | Phase 2 | 🔴 BLOCKED
4 | Verify coverage | 30 min | Phase 3 | 🔴 BLOCKED
5 | Validate config | 1 hour | Phase 2 | 🔴 BLOCKED
6 | Run & analyze bot | 1-2 hrs | Phase 4+5 | 🔴 BLOCKED

Total Timeline: 8-16 hours to production ready Critical Path: Tests → Fixes → Coverage → Validation Go/No-Go: After Phase 4 (coverage verification)

---

SUCCESS CRITERIA

✅ All tests passing

• All existing tests pass (currently have failures)
• No new test failures introduced
• Test output clean with no warnings

✅ Code coverage ≥ 80%

• Overall coverage ≥ 80% (will measure after fixes)
• All critical packages covered
• High-risk code paths covered

✅ Profitability validated

• Thresholds verified against market
• Gas estimation accurate
• Config settings documented

✅ Bot execution successful

• Binary builds without errors
• Bot starts without errors
• Bot detects opportunities
• Opportunity detection logged
• No unhandled panics

---

RISK MITIGATION

HIGH RISK: Test failures persist

Mitigation: Review git history, understand why tests fail, fix root cause

MEDIUM RISK: Coverage stays below 80%

Mitigation: Prioritize critical packages, implement coverage-driven testing

LOW RISK: Bot doesn't detect opportunities

Mitigation: Bot architecture is sound, likely just configuration tuning needed

---

TOOLS & COMMANDS REFERENCE

Running Tests

# Test single package
go test -v ./pkg/arbitrage

# Test all packages
go test -v -coverprofile=coverage.out ./pkg/... ./internal/...

# Check coverage
go tool cover -func=coverage.out | tail -1

# Generate HTML report
go tool cover -html=coverage.out -o coverage.html

Building Bot

# Normal build
make build

# Release build
make build-release

# In Podman
podman run -it --rm -v $(pwd):/app golang:1.25-alpine go build -o /app/bin/mev-bot ./cmd/mev-bot

Running Bot

# With logging
LOG_LEVEL=debug ./bin/mev-bot start

# With metrics
METRICS_ENABLED=true ./bin/mev-bot start

# With timeout for testing
timeout 300 ./bin/mev-bot start

---

NEXT IMMEDIATE STEPS

1. WAIT: For full test run to complete (currently running - bash ddf0fe)
2. ANALYZE: Check full test results and coverage report
3. PRIORITIZE: List failures by severity
4. FIX: Address high-severity failures first
5. ITERATE: Run tests after each fix, verify progress
6. VALIDATE: Ensure 80%+ coverage before moving to Phase 5

---

DECISION FRAMEWORK

If coverage < 50% after fixes: → Implement comprehensive test suite (8+ hours)

If coverage 50-80% after fixes: → Targeted testing for uncovered packages (2-4 hours)

If coverage > 80% after fixes: → Proceed to profitability validation and bot testing (2-3 hours)

---

PRODUCTION DEPLOYMENT CHECKLIST

Only deploy when ALL of these are complete:

• All tests passing (100% pass rate)
• Coverage ≥ 80% (documented in report)
• Profitability thresholds validated
• Bot successfully detects opportunities
• Opportunity execution working correctly
• Error handling verified
• Performance acceptable (< 1s latency)
• Logging working correctly
• Monitoring/metrics active
• Alerting configured
• Kill switches ready

---

Generated: 2025-11-06 Status: IN PROGRESS - Awaiting full test results Next Update: When test results available (bash ddf0fe completes)