mev-beta/docs/POOL_DATA_ERRORS_ROOT_CAUSE_ANALYSIS.md

Pool Data Errors - Root Cause Analysis & Fix Plan

Date: November 3, 2025 Status: Active Investigation Impact: High - Affecting opportunity executability validation

Executive Summary

Pool data errors are preventing the system from validating opportunities as executable. Currently, 347+ opportunities detected but 0 are marked as executable, all being rejected due to inability to fetch pool reserve data.

Key Finding

No opportunities are executable because pool validation is failing silently rather than returning proper error messages for filtering.

---

Root Cause Analysis

1. Primary Blocker: RPC Connection Failures

Evidence:

Post "https://arb1.arbitrum.io/rpc": dial tcp: lookup arb1.arbitrum.io: Temporary failure in name resolution

Impact:

• Primary RPC endpoint unreachable
• System falls back to fallback mode (basic block polling)
• Pool data cannot be fetched

Current Status: INTERMITTENT (was happening 12:00-12:12, recovered by 14:11)

---

2. Secondary: Batch Pool Data Fetch Timeouts

Evidence:

[WARN] Batch fetch failed for 0x42FC852A750BA93D5bf772ecdc857e87a86403a9:
       no data returned for pool - recording failure
[WARN] Failed to fetch batch 0-1: batch fetch V3 data failed:
       Post "https://arb1.arbitrum.io/rpc": context deadline exceeded

Root Cause:

• Batch fetcher using 10-second timeout
• Network latency + RPC overload = frequent timeouts
• Pools are being queried one-at-a-time instead of true batch

Affected Code: pkg/datafetcher/batch_fetcher.go

Impact:

• Legitimate pools failing due to timeout
• Same pools retried repeatedly (inefficient)
• Pools being blacklisted prematurely

---

3. Tertiary: Division by Zero in Smart Contracts

Evidence:

[WARN] Failed to fetch batch 0-1: batch fetch V3 data failed:
       execution reverted: division or modulo by zero

Root Causes:

• Querying uninitialized/zero-liquidity pools
• Non-standard pool implementations (broken fee() function, etc.)
• Smart contract state inconsistencies on L2

Affected Pools: ~10-15 pools (from 2025-11-02 logs)

---

4. Quaternary: Non-Standard Pool Implementations

Evidence:

[ERROR] Error getting pool data for 0xC6962004f452bE9203591991D15f6b388e09E8D0:
        pool ...is blacklisted: failed to call token1() - non-standard pool contract

Issue:

• Some pools don't follow standard ERC-20 interface
• token0(), token1() calls fail
• No graceful fallback to skip these pools

Current Handling: Blacklisting (correct), but error message suggests filtering could be better

---

Why All Opportunities Show "Not Executable"

Call Chain:

1. Swap event detected ✅
2. Opportunity analyzed ✅
3. Pool validation triggered for executability check
   • Attempts to fetch reserve data
   • RPC call fails or times out
   • Execution marked as false (default)
4. Opportunity logged with isExecutable:false

The Critical Issue:

When pool data can't be fetched, the system doesn't return proper error context for intelligent filtering. Instead, it:

• Returns nil reserves
• Marks as non-executable
• Doesn't distinguish between:
  • "Pool doesn't exist" (skip)
  • "RPC timeout" (retry)
  • "Non-standard pool" (blacklist)

---

System Status

Watch Script Output

• Opportunities Detected: 347+
• Executable: 0 (all failing pool validation)
• Executions: 0
• Errors: 0 (watch script filters out expected warnings)

Logs Status

2025/11/03 14:16:11 - Present
✅ Watch script successfully reading logs
✅ Opportunity detection working
❌ Pool validation blocking all executions

---

Solution Strategy

Phase 1: Immediate (Next 30 minutes)

1. Increase batch fetch timeout from 10s to 30s
2. Implement exponential backoff for retry logic
3. Add proper error context to distinguish error types

Phase 2: Short-term (Next hour)

1. Fix RPC endpoint configuration if primary is down
2. Implement batch caching to avoid repeated failures
3. Add pool pre-validation before RPC queries

Phase 3: Medium-term (Today)

1. Smart pool filtering - skip known bad contracts early
2. Improved monitoring - track pool failure patterns
3. Emergency fallback - use backup RPC providers

---

Affected Code Files

File	Issue	Priority
pkg/datafetcher/batch_fetcher.go	10s timeout, no backoff	HIGH
pkg/scanner/market/scanner.go	No error context in pool fetch	HIGH
pkg/scanner/market/pool_validator.go	Pre-validation could filter better	MEDIUM
pkg/uniswap/multicall.go	No fallback for failed calls	MEDIUM

---

Metrics to Track

• Pool fetch success rate (target: >95%)
• RPC timeout frequency (target: <1%)
• Pool blacklist size (current: ~10-15)
• Opportunity executability rate (current: 0%, target: >5%)

---

Next Actions

1. Read batch fetcher timeout configuration
2. Implement improved error handling
3. Add retry logic with backoff
4. Test with current opportunity stream
5. Monitor for improvement in executability rate