fix(multicall): resolve critical multicall parsing corruption issues

- Added comprehensive bounds checking to prevent buffer overruns in multicall parsing
- Implemented graduated validation system (Strict/Moderate/Permissive) to reduce false positives
- Added LRU caching system for address validation with 10-minute TTL
- Enhanced ABI decoder with missing Universal Router and Arbitrum-specific DEX signatures
- Fixed duplicate function declarations and import conflicts across multiple files
- Added error recovery mechanisms with multiple fallback strategies
- Updated tests to handle new validation behavior for suspicious addresses
- Fixed parser test expectations for improved validation system
- Applied gofmt formatting fixes to ensure code style compliance
- Fixed mutex copying issues in monitoring package by introducing MetricsSnapshot
- Resolved critical security vulnerabilities in heuristic address extraction
- Progress: Updated TODO audit from 10% to 35% complete

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

docs/8_reports/enhancements/arbitrage_executor_plan.md

@@ -0,0 +1,43 @@
+# Arbitrage Executor Hardening Plan
+
+## Overview
+Replace simulated flash-swap execution paths with on-chain integrations, accurate gas handling, and post-trade reconciliation. The current implementation relies on mock receipts, placeholder calldata, and synthetic pool data.
+
+## Tasks
+
+1. **Contract Integration**
+   - [ ] Add Go bindings for the flash swap/aggregator contract (generate via `abigen` if absent).
+   - [ ] Define canonical calldata struct matching the on-chain function signature (token path, pool info, slippage limits, recipient).
+
+2. **Transaction Submission**
+   - [x] Replace `submitTransaction` with actual contract call using flash swap aggregator binding.
+   - [ ] Implement nonce management and ensure transactions are signed with the configured key manager.
+
+3. **Gas Estimation & Pricing**
+   - [x] Swap placeholder gas limit with `client.EstimateGas` (include calldata and value).
+   - [x] Fetch base fee + priority tip dynamically; tie into Arbitrum fee/tip suggestions.
+   - [ ] Enforce config-driven max fee/tip thresholds (fallback to abort rather than cap to static value).
+
+4. **Confirmation & Reverts**
+   - [ ] Replace simulated receipt in `waitForConfirmation` with polling `TransactionReceipt`, respecting context timeout.
+   - [ ] Decode revert reasons/logs for diagnostics; handle dropped/replace-by-fee scenarios.
+
+5. **Profit/Loss Calculation**
+   - [ ] Update `calculateActualProfit` to parse emitted events/logs for actual amounts in/out.
+   - [ ] Subtract real gas cost from on-chain data; convert to `UniversalDecimal` with correct symbols.
+   - [ ] Emit structured logs/metrics for realized profit, slippage, and gas usage.
+
+6. **Fallback & Simulation**
+   - [ ] Move current simulation helpers into a `*_test.go` or dev-only path; guard with build tags if necessary.
+
+7. **Tests**
+   - [ ] Unit tests using mocked contract interface for success, revert, and timeout.
+   - [ ] Integration test with geth simulated backend or fork network verifying end-to-end flash swap.
+
+8. **Documentation**
+   - [ ] Update `docs/5_development/` and `docs/6_operations/` with new requirements (contract addresses, env vars).
+   - [ ] Note the removal of simulated paths; include troubleshooting steps for on-chain failures.
+
+References:
+- `pkg/arbitrage/executor.go` (simulateFlashSwapArbitrage, executeArbitrage)
+- `pkg/arbitrage/flash_executor.go` (submitTransaction, waitForConfirmation, calculateActualProfit)

docs/8_reports/enhancements/arbitrum_integration_plan.md

@@ -0,0 +1,28 @@
+# Arbitrum Integration Enhancement Plan
+
+## Goal
+Ensure all Arbitrum network integrations (sequencer monitor, ABI decoders, RPC failover) are production-grade and resilient.
+
+## Tasks
+
+1. **Sequencer & Parser**
+   - [ ] Validate `monitor.ArbitrumMonitor` handles all transaction types; add tests with real block samples.
+   - [ ] Remove placeholder logging in `createArbitrumMonitor`, confirm default path uses full sequencer reader.
+
+2. **ABI Decoding**
+   - [ ] Expand coverage in `pkg/arbitrum/parser` for multicall variants, custom protocols.
+   - [ ] Add fuzz/property tests for decoder correctness.
+
+3. **Connection Management**
+   - [ ] Audit `pkg/arbitrum/connection.go` for failover, rate limiting, and health checks.
+   - [ ] Document required environment variables (`ARBITRUM_RPC_ENDPOINT`, `ARBITRUM_WS_ENDPOINT`), default policies.
+
+4. **Logging & Metrics**
+   - [ ] Ensure errors and reconnection attempts emit structured logs and Prometheus metrics.
+
+5. **Docs**
+   - [ ] Update `docs/6_operations/DEPLOYMENT_GUIDE.md` with operator steps for Arbitrum endpoints.
+
+## References
+- `pkg/arbitrum`, `pkg/monitor`
+- `docs/6_operations`

docs/8_reports/enhancements/cicd_plan.md

@@ -0,0 +1,26 @@
+# CI/CD Pipeline Enhancement Plan
+
+## Goal
+Ensure environment-specific pipelines cover all quality gates (lint, tests, security, profitability) and reflect production requirements. Legacy GitHub workflows remain manual-only fallbacks.
+
+## Tasks
+
+1. **Pipeline Coverage**
+   - [ ] Verify new workflows (`.drone.yml` pipelines `test-suite`, `security-suite`, `integration-opt-in`, plus `harness/pipelines/staging.yaml`) run appropriate targets for each environment.
+   - [ ] Add checks for gofmt, go mod tidy, unit/integration tests, math audit, profit simulation, security scans.
+
+2. **Cache & Secrets**
+   - [ ] Ensure caches do not leak secrets; ensure Drone/Harness runners mount secrets from the appropriate secret managers.
+   - [ ] Provide documentation for self-hosted runners if needed.
+
+3. **Artifact Retention**
+   - [ ] Upload math audit, simulation, coverage, and deployment summaries for every run.
+   - [ ] Set retention policies and naming conventions.
+
+4. **Docs**
+   - [ ] Update `docs/5_development/TESTING_BENCHMARKING.md` with pipeline descriptions.
+   - [ ] Add troubleshooting section for common CI failures.
+
+## References
+- `.drone.yml`, `harness/pipelines/`
+- `Makefile`, scripts invoked by CI

docs/8_reports/enhancements/database_plan.md

@@ -0,0 +1,30 @@
+# Database Layer Hardening Plan
+
+## Goal
+Ensure database access patterns are efficient, resilient, and match schema constraints across `pkg/database` and integrations.
+
+## Tasks
+
+1. **Schema & Migrations**
+   - [ ] Review current schema/migrations (if any) to confirm tables/indexes align with access patterns.
+   - [ ] Document schema in `docs/` (ER diagram, migration history).
+
+2. **Connection Management**
+   - [ ] Validate pool settings (`max_open`, `max_idle`, `conn max lifetime`) and expose via config.
+   - [ ] Implement health checks and failover for primary/replica setups.
+
+3. **Transaction Integrity**
+   - [ ] Ensure critical writes use transactions with rollback on error.
+   - [ ] Add tests for idempotent persistence (e.g., duplicate opportunities).
+
+4. **Performance & Monitoring**
+   - [ ] Instrument slow query logging and Prometheus metrics.
+   - [ ] Run load tests to ensure queue/execution writes keep up under peak load.
+
+5. **Docs & Runbooks**
+   - [ ] Update deployment docs with connection strings, maintenance tasks, backup/restore procedures.
+
+## References
+- `pkg/database/database.go`
+- `pkg/database/database_test.go`
+- `docs/6_operations/DEPLOYMENT_GUIDE.md`

docs/8_reports/enhancements/deployment_scripts_plan.md

@@ -0,0 +1,28 @@
+# Deployment & Scripts Hardening Plan
+
+## Goal
+Replace placeholder secrets, document secure deployment paths, and ensure scripts support staging/production parity.
+
+## Tasks
+
+1. **Secrets Management**
+   - [ ] Remove/test configs referencing placeholder keys (`MEV_BOT_ENCRYPTION_KEY`, keystore paths) and integrate with secret manager (Vault/SSM/etc.).
+   - [ ] Update scripts to fetch secrets securely (env injection, sealed secrets).
+
+2. **Run/Deploy Scripts**
+   - [ ] Audit `scripts/run.sh`, `deploy-*.sh` for hard-coded endpoints; parameterise via config.
+   - [ ] Add sanity checks before destructive actions (backups, diff of config).
+
+3. **Docker/Compose**
+   - [ ] Ensure compose files align with production environment (volumes, ports, metrics).
+   - [ ] Provide sample `.env.production` with guidance on required variables.
+
+4. **Rollback & Recovery**
+   - [ ] Document rollback procedures, data backup steps, and log locations for investigation.
+
+5. **Testing**
+   - [ ] Add lint/checks for scripts (shellcheck) in CI.
+
+## References
+- `scripts/`, `docker-compose.*`
+- `docs/6_operations/DEPLOYMENT_GUIDE.md`

docs/8_reports/enhancements/detection_engine_plan.md

@@ -0,0 +1,33 @@
+# Detection Engine Hardening Plan
+
+## Objectives
+Ensure the detection pipeline produces actionable opportunities with production-grade heuristics and hands them off to the execution stack automatically. Current implementation primarily logs opportunities and uses simplified scoring parameters.
+
+## Action Items
+
+1. **Opportunity Scoring & Filtering**
+   - [ ] Define canonical scoring function (profitability, confidence, liquidity, latency) aligned with business thresholds.
+   - [ ] Replace ad-hoc ROI/price-impact checks with configurable policies (pull from config/env).
+   - [ ] Add regression tests covering borderline cases (low confidence, high slippage, gas-constrained opportunities).
+
+2. **Execution Handoff**
+   - [x] Implement direct handoff to `ArbitrageExecutor` via callback (see `SetOpportunityHandler`).
+   - [ ] Ensure backpressure handling (bounded queue, metrics when saturated).
+   - [ ] Introduce retry/backoff for transient execution failures.
+
+3. **Concurrency & Worker Pools**
+   - [ ] Review `WorkerPool` sizing, ensure graceful shutdown and error propagation.
+   - [ ] Add instrumentation (metrics/logs) for scan time, queue depth, dropped opportunities.
+
+4. **Historical Validation**
+   - [ ] Replay recorded blocks/datasets to validate hit rate and false positives.
+   - [ ] Store misclassified opportunities for manual review.
+
+5. **Configuration & Docs**
+   - [ ] Centralise detection thresholds in config files (`config/*.yaml`) and document in `docs/5_development/CONFIGURATION.md`.
+   - [ ] Update runbooks to describe detection tuning knobs and alerting thresholds.
+
+## References
+- `pkg/arbitrage/detection_engine.go`
+- `pkg/arbitrage/service.go` (execution integration)
+- `pkg/metrics` for instrumentation

docs/8_reports/enhancements/documentation_plan.md

@@ -0,0 +1,24 @@
+# Documentation & Runbooks Enhancement Plan
+
+## Goal
+Ensure operational and development documentation reflects the current architecture, tooling, and audit requirements.
+
+## Tasks
+
+1. **Docs Audit**
+   - [ ] Review `docs/` tree for outdated instructions (e.g., placeholders, legacy scripts).
+   - [ ] Align README and quick-start guides with new pipelines and simulation tooling.
+
+2. **Operational Runbooks**
+   - [ ] Update `docs/6_operations/DEPLOYMENT_GUIDE.md` with full staging/production steps, including metrics, alerts, rollback.
+   - [ ] Add incident response/triage guides for risk breaches, sequencer downtime, failed flash swaps.
+
+3. **Audit Reports**
+   - [ ] Maintain `docs/8_reports/` with latest readiness, audit, and enhancement plans.
+   - [ ] Automate generation/links from CI artifacts where possible.
+
+4. **Agent Guides**
+   - [ ] Ensure `.qwen/`, `.claude/`, `.gemini/` guides reference updated commands and checklists.
+
+## References
+- `docs/`, `.qwen/`, `.claude/`

docs/8_reports/enhancements/execution_orchestrator_plan.md

@@ -0,0 +1,25 @@
+# Execution Queue & Orchestrator Plan
+
+## Goal
+Ensure the execution queue, orchestrator, and worker pools manage throughput reliably with visibility into performance under load.
+
+## Tasks
+
+1. **Queue Semantics**
+   - [ ] Review `pkg/execution/queue.go` to confirm FIFO/LIFO behaviour matches requirements.
+   - [ ] Add bounds and backpressure (drop/park behaviour) and document policies.
+
+2. **Worker Pool Management**
+   - [ ] Audit `pkg/orchestrator/coordinator.go` for race conditions, proper shutdown, and retry logic.
+   - [ ] Add tests simulating burst loads, slow executions, and worker failures.
+
+3. **Metrics & Logging**
+   - [ ] Instrument queue depth, processing latency, retry counts via Prometheus.
+   - [ ] Ensure structured logs highlight queue saturation and dropped tasks.
+
+4. **Configuration**
+   - [ ] Expose worker/queue sizing via config (`ExecutionConfig`), document defaults and tuning guidance.
+
+## References
+- `pkg/execution/queue.go`
+- `pkg/orchestrator/coordinator.go`

docs/8_reports/enhancements/market_scanner_plan.md

@@ -0,0 +1,29 @@
+# Market Data & Scanner Enhancement Plan
+
+## Goal
+Harden market scanning, sequencing, and opportunity discovery layers to ensure concurrency safety, RPC resilience, and realistic opportunity generation.
+
+## Tasks
+
+1. **Concurrency Review**
+   - [ ] Audit `pkg/scanner/concurrent.go`, `pkg/market` to ensure worker pools handle backpressure and shutdown gracefully.
+   - [ ] Add tests for race conditions and high-volume bursts.
+
+2. **RPC Failover & Monitoring**
+   - [ ] Validate `pkg/monitor` and `pkg/arbitrum` connection managers reuse failover lists, exponential backoff, and circuit breakers.
+   - [ ] Instrument metrics for RPC errors, reconnections, and backlog length.
+
+3. **Sequencer Integration**
+   - [ ] Replace fallback block polling with production-grade sequencer readers (ensure `createArbitrumMonitor` is default path).
+   - [ ] Confirm parser handles all transaction types (EIP-1559, L1->L2 messages) with tests using captured data.
+
+4. **Opportunity Quality**
+   - [ ] Validate detection heuristics against real block data (false positives/negatives).
+   - [ ] Integrate opportunity deduplication/deconfliction across multiple scanners.
+
+5. **Documentation**
+   - [ ] Update operational docs covering monitoring endpoints, scaling knobs, and troubleshooting.
+
+## References
+- `pkg/scanner`, `pkg/market`, `pkg/monitor`
+- `pkg/arbitrum` connection & parser files

docs/8_reports/enhancements/math_core_plan.md

@@ -0,0 +1,34 @@
+# Math & Pricing Core Hardening Plan
+
+## Objective
+Replace "simplified" math implementations with production-accurate formulas across all supported DEXes (Uniswap V2/V3, Camelot/Algebra, Curve, Balancer, etc.), backed by deterministic tests and documentation.
+
+## Tasks
+
+1. **Inventory Simplified Functions**
+   - [ ] Enumerate all locations tagged as simplified (e.g., `pkg/math/dex_math.go`, `exchange_pricing.go`, `price_impact.go`, `arbitrage_calculator.go`).
+   - [ ] Cross-reference official DEX specs/whitepapers for correct formulas.
+
+2. **Implement Accurate Formulas**
+   - [ ] Uniswap V2/V3 constant-product and concentrated liquidity calculations (ticks, liquidity changes, fee tiers).
+   - [ ] Curve stable swap invariant (iterative solver) and fee handling.
+   - [ ] Balancer weighted pool math and spot price impact.
+   - [ ] Algebra/Camelot specific fees/liquidity adjustments.
+
+3. **Precision & Decimal Handling**
+   - [ ] Ensure `UniversalDecimal` usage preserves precision boundaries, rounding modes documented in `docs/MATH_OPTIMIZATIONS.md`.
+   - [ ] Validate conversions against property tests (`test/property/pricing_property_test.go`).
+
+4. **Testing & Benchmarks**
+   - [ ] Expand deterministic vectors in `tools/math-audit` to cover new formulas.
+   - [ ] Add targeted unit/property tests for each new implementation.
+   - [ ] Benchmark key functions to confirm performance is acceptable and document deltas.
+
+5. **Documentation**
+   - [ ] Update math design docs (`docs/MATH_OPTIMIZATIONS.md`, `docs/5_development/TESTING_BENCHMARKING.md`) with formula derivations and usage guidance.
+   - [ ] Provide example calculations for auditors.
+
+## References
+- `pkg/math/dex_math.go`, `pkg/math/exchange_pricing.go`, `pkg/math/exchange_math.go`
+- `pkg/uniswap/pricing.go`, `pkg/pools/...`
+- `tools/math-audit`

docs/8_reports/enhancements/metrics_plan.md

@@ -0,0 +1,25 @@
+# Metrics & Telemetry Enhancement Plan
+
+## Goal
+Expand metrics coverage for profitability, latency, and error conditions and ensure dashboards/alerts align with SRE expectations.
+
+## Tasks
+
+1. **Metric Inventory**
+   - [ ] Catalogue existing metrics in `pkg/metrics/metrics.go` and identify gaps (profit factor, queue depth, RPC errors).
+   - [ ] Ensure every critical subsystem records Prometheus metrics.
+
+2. **Alerting & Dashboards**
+   - [ ] Update Grafana dashboards to include new metrics; document recommended alert thresholds.
+   - [ ] Integrate simulation outputs to set baseline expectations for hit rate and profit.
+
+3. **Endpoint Hardening**
+   - [ ] Validate authentication/IP allowlist for `/metrics` endpoint; support TLS/ingress integration.
+   - [ ] Add health checks for metrics server (liveness/readiness signals).
+
+4. **Documentation**
+   - [ ] Extend `docs/6_operations/DEPLOYMENT_GUIDE.md` with monitoring instructions and alert-response runbooks.
+
+## References
+- `pkg/metrics/metrics.go`
+- `monitoring/prometheus.yml`, Grafana configs

docs/8_reports/enhancements/risk_management_plan.md

@@ -0,0 +1,31 @@
+# Risk Management Enhancement Plan
+
+## Goal
+Align risk controls with production policy: configurable thresholds, historical performance feedback, and comprehensive metrics.
+
+## Tasks
+
+1. **Configuration Externalisation**
+   - [ ] Move hard-coded limits (max position size, daily loss, profit threshold, gas caps) into config/environment.
+   - [ ] Support per-market overrides (e.g., by token pair or exchange).
+
+2. **Dynamic Scoring**
+   - [ ] Revisit `calculateRiskScore`, `calculateConfidence`, `calculateProfitability` to include real-time volatility, slippage history, and order book depth.
+   - [ ] Integrate historical trade outcomes (from database) to adjust thresholds dynamically.
+
+3. **Circuit Breaker & Monitoring**
+   - [ ] Ensure circuit breaker triggers persist across restarts and expose Prometheus metrics for activations.
+   - [ ] Add alerting thresholds (e.g., consecutive failures, daily loss approaching limit).
+
+4. **Testing**
+   - [ ] Add integration tests verifying risk rejections for low profit, high slippage, excessive gas price with configurable values.
+   - [ ] Include fuzz/property tests for risk-related calculations.
+
+5. **Documentation**
+   - [ ] Update `docs/5_development/CONFIGURATION.md` with risk knobs.
+   - [ ] Provide runbook entries for adjusting risk parameters during incidents.
+
+## References
+- `pkg/risk/manager.go`
+- `pkg/arbitrage/service.go` (risk integration)
+- `pkg/metrics` for monitoring hooks

docs/8_reports/enhancements/simulation_tools_plan.md

@@ -0,0 +1,33 @@
+# Simulation Tools Enhancement Plan
+
+## Goal
+Align the profitability replay harness and embedded simulation helpers with production requirements: real historical data, accurate fee/slippage modelling, and automated validation in CI.
+
+## Tasks
+
+1. **Historical Data Ingestion**
+   - [ ] Define vector schema for real opportunities (timestamp, DEX, expected vs. realized profit, route data).
+   - [ ] Build extraction pipeline from historical archive/DB to `tools/simulation/vectors/` (with sanitised examples).
+   - [ ] Add versioning/checksums for vectors used in CI.
+
+2. **Simulation Accuracy**
+   - [ ] Replace hard-coded fees/reserves (`tools/simulation/main.go`) with per-exchange metadata.
+   - [ ] Model slippage, gas costs, and failed trades using observed distributions.
+   - [ ] Introduce configuration for latency, backrunning competition, gas spikes.
+
+3. **Integration with Arbitrage Pipeline**
+   - [ ] Provide CLI option to replay directly against `ArbitrageService` / executor in dry-run mode.
+   - [ ] Emit metrics for hit rate and net profit and compare against Prometheus data.
+
+4. **CI Automation**
+   - [ ] Add `make simulate-profit` to staging/audit workflows with artifact retention.
+   - [ ] Set thresholds (e.g., net profit >0, hit rate >50%) to fail CI when regressions occur.
+
+5. **Documentation & Runbooks**
+   - [ ] Update `docs/5_development/TESTING_BENCHMARKING.md` and `docs/6_operations/DEPLOYMENT_GUIDE.md` with guidance on capturing vectors and interpreting reports.
+   - [ ] Provide runbook entry for simulation dashboards and alerting.
+
+## References
+- `tools/simulation/main.go`
+- `scripts/run_profit_simulation.sh`
+- `pkg/arbitrage` simulation helper functions