feat: create v2-prep branch with comprehensive planning
Restructured project for V2 refactor: **Structure Changes:** - Moved all V1 code to orig/ folder (preserved with git mv) - Created docs/planning/ directory - Added orig/README_V1.md explaining V1 preservation **Planning Documents:** - 00_V2_MASTER_PLAN.md: Complete architecture overview - Executive summary of critical V1 issues - High-level component architecture diagrams - 5-phase implementation roadmap - Success metrics and risk mitigation - 07_TASK_BREAKDOWN.md: Atomic task breakdown - 99+ hours of detailed tasks - Every task < 2 hours (atomic) - Clear dependencies and success criteria - Organized by implementation phase **V2 Key Improvements:** - Per-exchange parsers (factory pattern) - Multi-layer strict validation - Multi-index pool cache - Background validation pipeline - Comprehensive observability **Critical Issues Addressed:** - Zero address tokens (strict validation + cache enrichment) - Parsing accuracy (protocol-specific parsers) - No audit trail (background validation channel) - Inefficient lookups (multi-index cache) - Stats disconnection (event-driven metrics) Next Steps: 1. Review planning documents 2. Begin Phase 1: Foundation (P1-001 through P1-010) 3. Implement parsers in Phase 2 4. Build cache system in Phase 3 5. Add validation pipeline in Phase 4 6. Migrate and test in Phase 5 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
Administrator
325
tests/calculation-validation/README.md
Normal file
325
tests/calculation-validation/README.md
Normal file
@@ -0,0 +1,325 @@
|
||||
# Arbitrage Calculation Validation Framework
|
||||
|
||||
This testing framework extracts arbitrage opportunity data from MEV bot logs and validates profit calculations to ensure accuracy and detect bugs.
|
||||
|
||||
## Overview
|
||||
|
||||
The framework consists of three main components:
|
||||
|
||||
1. **Log Extraction** (`scripts/test-calculations.sh`) - Extracts opportunities, calculations, and threshold checks from logs
|
||||
2. **Validation Library** (Go packages) - Parses and validates calculation correctness
|
||||
3. **Replay Tool** (`replay.go`) - Replays calculations and generates validation reports
|
||||
|
||||
## Quick Start
|
||||
|
||||
### Extract and Analyze Logs
|
||||
|
||||
```bash
|
||||
# Extract from latest container logs
|
||||
./scripts/test-calculations.sh
|
||||
|
||||
# Extract from specific log file
|
||||
./scripts/test-calculations.sh /path/to/logs.txt
|
||||
```
|
||||
|
||||
### Run Validation Tests
|
||||
|
||||
```bash
|
||||
# Run unit tests
|
||||
go test ./tests/calculation-validation/... -v
|
||||
|
||||
# Or in container
|
||||
podman exec mev-bot-dev-master-dev go test ./tests/calculation-validation/... -v
|
||||
```
|
||||
|
||||
### Replay Calculations
|
||||
|
||||
```bash
|
||||
# Replay and validate all calculations
|
||||
go run ./tests/calculation-validation/replay.go
|
||||
|
||||
# With verbose output
|
||||
go run ./tests/calculation-validation/replay.go -verbose
|
||||
|
||||
# Custom test directory
|
||||
go run ./tests/calculation-validation/replay.go -dir tests/calculation-validation
|
||||
```
|
||||
|
||||
## What It Validates
|
||||
|
||||
### ✅ Profit Threshold Checks
|
||||
|
||||
Validates that opportunities are correctly marked as executable based on profit thresholds:
|
||||
|
||||
```go
|
||||
netProfit >= minThreshold → executable = true
|
||||
netProfit < minThreshold → executable = false
|
||||
```
|
||||
|
||||
**Critical Bug Fixed:** Before the fix, the code was comparing the integer part of ETH (834) to wei threshold (100000000000000), causing all opportunities to be rejected.
|
||||
|
||||
**After Fix:** Properly converts both values to same units (ETH) for comparison:
|
||||
```go
|
||||
834.210302 ETH >= 0.0001 ETH → EXECUTABLE ✅
|
||||
```
|
||||
|
||||
### ✅ ETH to Wei Conversions
|
||||
|
||||
Validates that ETH amounts are correctly converted to wei:
|
||||
|
||||
```go
|
||||
1 ETH = 1,000,000,000,000,000,000 wei (1e18)
|
||||
```
|
||||
|
||||
### ✅ V3 Swap Calculations
|
||||
|
||||
Validates Uniswap V3 swap calculations including:
|
||||
- Fee deductions (0.05%, 0.3%, 1%)
|
||||
- Output amounts
|
||||
- Zero output detection
|
||||
|
||||
### ✅ Opportunity Consistency
|
||||
|
||||
Validates logical consistency:
|
||||
- Executable opportunities have positive profit
|
||||
- Executable opportunities meet minimum threshold
|
||||
- Rejected opportunities have valid reject reasons
|
||||
|
||||
## Output Files
|
||||
|
||||
### Extraction Phase
|
||||
|
||||
```
|
||||
tests/calculation-validation/
|
||||
├── extracted/
|
||||
│ ├── executable_opportunities.log # Opportunities marked executable
|
||||
│ ├── opportunity_details.log # Full opportunity records
|
||||
│ ├── v3_calculations.log # V3 swap calculations
|
||||
│ ├── threshold_checks.log # Profit threshold validations
|
||||
│ ├── rejections.log # Rejected opportunities
|
||||
│ ├── profit_values.txt # Extracted profit amounts
|
||||
│ └── test_data.json # Structured test data
|
||||
└── reports/
|
||||
└── validation_report_*.md # Summary validation report
|
||||
```
|
||||
|
||||
## Example Report
|
||||
|
||||
```
|
||||
═══════════════════════════════════════════════════════════════════════
|
||||
Extraction Complete!
|
||||
═══════════════════════════════════════════════════════════════════════
|
||||
|
||||
Summary Statistics:
|
||||
✅ Executable Opportunities: 11
|
||||
📊 Total Opportunity Records: 23
|
||||
🔢 V3 Calculations: 600
|
||||
✔️ Threshold Checks: 11
|
||||
❌ Rejections: 23
|
||||
|
||||
Profit Statistics:
|
||||
💰 Total Profit: 4,053.69 ETH
|
||||
📈 Average Profit: 368.517 ETH
|
||||
🔝 Max Profit: 1,363.86 ETH
|
||||
📉 Min Profit: 0.003708 ETH
|
||||
```
|
||||
|
||||
## Validation Results
|
||||
|
||||
After running the replay tool, you'll get:
|
||||
|
||||
```
|
||||
━━━ Profit Threshold Checks ━━━
|
||||
Total: 11
|
||||
✅ Valid: 11
|
||||
❌ Invalid: 0
|
||||
Success Rate: 100.00%
|
||||
|
||||
━━━ Critical Bug Fix Validation ━━━
|
||||
Testing bug fix scenario:
|
||||
Net Profit: 834.210302 ETH
|
||||
Min Threshold: 0.0001 ETH
|
||||
✅ BUG FIX VALIDATED: Correctly marked as executable
|
||||
|
||||
Testing edge case (profit == threshold):
|
||||
Net Profit: 0.0001 ETH
|
||||
✅ Edge case handled correctly
|
||||
|
||||
Testing rejection case (profit < threshold):
|
||||
Net Profit: 0.00001 ETH
|
||||
✅ Correctly rejected
|
||||
```
|
||||
|
||||
## Test Coverage
|
||||
|
||||
The framework includes comprehensive unit tests:
|
||||
|
||||
### `validator_test.go`
|
||||
|
||||
- ✅ `TestValidateThresholdCheck` - Threshold validation logic
|
||||
- ✅ `TestValidateThresholdComparison` - Critical bug fix validation
|
||||
- ✅ `TestValidateV3Calculation` - Uniswap V3 math
|
||||
- ✅ `TestCompareETHtoWei` - Unit conversion
|
||||
- ✅ `TestCalculateStatistics` - Profit statistics
|
||||
- ✅ `TestValidateOpportunity` - Full opportunity validation
|
||||
|
||||
Run tests:
|
||||
```bash
|
||||
go test ./tests/calculation-validation/... -v -cover
|
||||
```
|
||||
|
||||
## Architecture
|
||||
|
||||
### Data Flow
|
||||
|
||||
```
|
||||
┌─────────────────┐
|
||||
│ Log Files │
|
||||
└────────┬────────┘
|
||||
│
|
||||
▼
|
||||
┌─────────────────┐
|
||||
│ Log Parser │ (parser.go)
|
||||
│ - Regex │
|
||||
│ - Extraction │
|
||||
└────────┬────────┘
|
||||
│
|
||||
▼
|
||||
┌─────────────────┐
|
||||
│ Test Data │ (types.go)
|
||||
│ - Opportunities│
|
||||
│ - Calculations │
|
||||
│ - Thresholds │
|
||||
└────────┬────────┘
|
||||
│
|
||||
▼
|
||||
┌─────────────────┐
|
||||
│ Validator │ (validator.go)
|
||||
│ - Comparison │
|
||||
│ - Validation │
|
||||
└────────┬────────┘
|
||||
│
|
||||
▼
|
||||
┌─────────────────┐
|
||||
│ Test Report │
|
||||
│ - Statistics │
|
||||
│ - Errors │
|
||||
│ - Warnings │
|
||||
└─────────────────┘
|
||||
```
|
||||
|
||||
## Use Cases
|
||||
|
||||
### 1. Verify Bug Fixes
|
||||
|
||||
After applying a fix to profit calculations, run the framework to validate:
|
||||
|
||||
```bash
|
||||
./scripts/test-calculations.sh
|
||||
go run ./tests/calculation-validation/replay.go -verbose
|
||||
```
|
||||
|
||||
### 2. Continuous Integration
|
||||
|
||||
Add to CI pipeline to catch calculation regressions:
|
||||
|
||||
```bash
|
||||
#!/bin/bash
|
||||
# Get logs from test run
|
||||
./mev-bot start &
|
||||
BOT_PID=$!
|
||||
sleep 60
|
||||
kill $BOT_PID
|
||||
|
||||
# Extract and validate
|
||||
./scripts/test-calculations.sh logs/mev_bot.log
|
||||
go test ./tests/calculation-validation/... || exit 1
|
||||
```
|
||||
|
||||
### 3. Performance Analysis
|
||||
|
||||
Extract profit statistics over time:
|
||||
|
||||
```bash
|
||||
for log in logs/archive/*.log; do
|
||||
./scripts/test-calculations.sh "$log"
|
||||
done
|
||||
|
||||
# Aggregate results
|
||||
find tests/calculation-validation/reports -name "*.md" -exec cat {} \;
|
||||
```
|
||||
|
||||
## Critical Metrics
|
||||
|
||||
The framework tracks key metrics to ensure bot health:
|
||||
|
||||
| Metric | Healthy Range | Warning | Critical |
|
||||
|--------|---------------|---------|----------|
|
||||
| Executable % | > 5% | 1-5% | < 1% |
|
||||
| Threshold Pass Rate | 100% | 95-99% | < 95% |
|
||||
| V3 Zero Outputs | < 10% | 10-25% | > 25% |
|
||||
| Avg Profit | > 0.1 ETH | 0.01-0.1 ETH | < 0.01 ETH |
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### No Executable Opportunities Found
|
||||
|
||||
**Symptom:** `✓ Found 0 executable opportunities`
|
||||
|
||||
**Possible Causes:**
|
||||
1. Log file is from before the bug fix was applied
|
||||
2. Bot isn't detecting any profitable opportunities
|
||||
3. All opportunities are being rejected
|
||||
|
||||
**Solution:**
|
||||
```bash
|
||||
# Get fresh logs
|
||||
podman logs mev-bot-dev-master-dev 2>&1 > /tmp/fresh_logs.txt
|
||||
./scripts/test-calculations.sh /tmp/fresh_logs.txt
|
||||
|
||||
# Check for threshold validation
|
||||
grep "Profit threshold check" /tmp/fresh_logs.txt
|
||||
```
|
||||
|
||||
### Tests Failing
|
||||
|
||||
**Symptom:** `go test` failures
|
||||
|
||||
**Solution:**
|
||||
```bash
|
||||
# Check if modules are initialized
|
||||
cd tests/calculation-validation
|
||||
go mod init mev-bot/tests/calculation-validation 2>/dev/null || true
|
||||
go mod tidy
|
||||
|
||||
# Run with verbose output
|
||||
go test -v
|
||||
```
|
||||
|
||||
### Extraction Script Not Found
|
||||
|
||||
**Symptom:** `bash: ./scripts/test-calculations.sh: Permission denied`
|
||||
|
||||
**Solution:**
|
||||
```bash
|
||||
chmod +x ./scripts/test-calculations.sh
|
||||
```
|
||||
|
||||
## Future Enhancements
|
||||
|
||||
- [ ] Real-time validation during bot operation
|
||||
- [ ] Alerting on calculation anomalies
|
||||
- [ ] Historical trend analysis
|
||||
- [ ] Multi-DEX calculation validation
|
||||
- [ ] Gas cost accuracy validation
|
||||
- [ ] Slippage calculation validation
|
||||
|
||||
## Related Documentation
|
||||
|
||||
- [Bug Fix Solution](/docker/mev-beta/logs/BUG_FIX_SOLUTION_20251109.md) - Detailed bug analysis
|
||||
- [Critical Bugs Found](/docker/mev-beta/logs/CRITICAL_BUGS_FOUND_20251109.md) - Initial bug discovery
|
||||
- [Profit Calculator](/docker/mev-beta/pkg/profitcalc/profit_calc.go) - Source code
|
||||
|
||||
## License
|
||||
|
||||
Part of MEV Bot project - see main LICENSE file.
|
||||
Reference in New Issue
Block a user