fix(multicall): resolve critical multicall parsing corruption issues
- Added comprehensive bounds checking to prevent buffer overruns in multicall parsing - Implemented graduated validation system (Strict/Moderate/Permissive) to reduce false positives - Added LRU caching system for address validation with 10-minute TTL - Enhanced ABI decoder with missing Universal Router and Arbitrum-specific DEX signatures - Fixed duplicate function declarations and import conflicts across multiple files - Added error recovery mechanisms with multiple fallback strategies - Updated tests to handle new validation behavior for suspicious addresses - Fixed parser test expectations for improved validation system - Applied gofmt formatting fixes to ensure code style compliance - Fixed mutex copying issues in monitoring package by introducing MetricsSnapshot - Resolved critical security vulnerabilities in heuristic address extraction - Progress: Updated TODO audit from 10% to 35% complete 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
Krypto Kajun
379
docs/5_development/CI_PIPELINE_USAGE.md
Normal file
379
docs/5_development/CI_PIPELINE_USAGE.md
Normal file
@@ -0,0 +1,379 @@
|
||||
# CI/CD Pipeline Usage Guide
|
||||
|
||||
## 🚀 Quick Reference
|
||||
|
||||
### Make Commands (Recommended)
|
||||
```bash
|
||||
# Pre-commit validation (fastest - 10-30 seconds)
|
||||
make ci-precommit
|
||||
|
||||
# Quick CI pipeline (30-60 seconds)
|
||||
make ci-quick
|
||||
|
||||
# Development CI pipeline (1-2 minutes)
|
||||
make ci-dev
|
||||
|
||||
# Full CI pipeline (3-5 minutes)
|
||||
make ci-full
|
||||
|
||||
# Container-based CI
|
||||
make ci-container
|
||||
make ci-container-quick
|
||||
|
||||
# Watch mode (auto-run on file changes)
|
||||
make ci-watch
|
||||
make ci-watch-quick
|
||||
```
|
||||
|
||||
### Direct Script Usage
|
||||
```bash
|
||||
# Individual scripts
|
||||
./scripts/ci-precommit.sh
|
||||
./scripts/ci-quick.sh
|
||||
./scripts/ci-dev.sh
|
||||
./scripts/ci-full.sh
|
||||
./scripts/ci-container.sh [quick|dev|full]
|
||||
./scripts/ci-watch.sh [quick|precommit]
|
||||
```
|
||||
|
||||
## 📋 Pipeline Types Explained
|
||||
|
||||
### 1. Pre-commit (`make ci-precommit`)
|
||||
**Purpose**: Fastest validation before committing code
|
||||
**Time**: 10-30 seconds
|
||||
**What it does**:
|
||||
- ✅ Build binary
|
||||
- ✅ Run unit tests
|
||||
- ✅ Check code formatting
|
||||
- ✅ Static analysis (go vet)
|
||||
|
||||
```bash
|
||||
# Use before every commit
|
||||
make ci-precommit
|
||||
```
|
||||
|
||||
### 2. Quick CI (`make ci-quick`)
|
||||
**Purpose**: Fast comprehensive validation
|
||||
**Time**: 30-60 seconds
|
||||
**What it does**:
|
||||
- ✅ All pre-commit checks
|
||||
- ✅ Advanced linting
|
||||
- ❌ Skips: Docker, Math Audit, Security scans
|
||||
|
||||
```bash
|
||||
# Good for regular development
|
||||
make ci-quick
|
||||
```
|
||||
|
||||
### 3. Development CI (`make ci-dev`)
|
||||
**Purpose**: Balanced speed and coverage
|
||||
**Time**: 1-2 minutes
|
||||
**What it does**:
|
||||
- ✅ All quick CI checks
|
||||
- ✅ Math audit validation
|
||||
- ✅ Security scanning
|
||||
- ❌ Skips: Docker build only
|
||||
|
||||
```bash
|
||||
# Best for daily development workflow
|
||||
make ci-dev
|
||||
```
|
||||
|
||||
### 4. Full CI (`make ci-full`)
|
||||
**Purpose**: Complete validation for releases
|
||||
**Time**: 3-5 minutes
|
||||
**What it does**:
|
||||
- ✅ Everything enabled
|
||||
- ✅ Docker build and validation
|
||||
- ✅ Complete security audit
|
||||
- ✅ Mathematical validation
|
||||
- ✅ All reports generated
|
||||
|
||||
```bash
|
||||
# Use before releases or PR creation
|
||||
make ci-full
|
||||
```
|
||||
|
||||
## 🐳 Container-based CI
|
||||
|
||||
### When to Use Container CI
|
||||
- Isolation from host environment
|
||||
- Consistent build environment
|
||||
- Testing without local Go installation
|
||||
- CI/CD server deployment
|
||||
|
||||
### Container Commands
|
||||
```bash
|
||||
# Development CI in container
|
||||
make ci-container
|
||||
|
||||
# Quick validation in container
|
||||
make ci-container-quick
|
||||
|
||||
# Direct script with options
|
||||
./scripts/ci-container.sh quick # Fast
|
||||
./scripts/ci-container.sh dev # Balanced
|
||||
./scripts/ci-container.sh full # Complete (no Docker)
|
||||
```
|
||||
|
||||
### Container Requirements
|
||||
```bash
|
||||
# Ensure container runtime is available
|
||||
podman --version # or docker --version
|
||||
|
||||
# Install if needed
|
||||
sudo apt install podman # or docker.io
|
||||
```
|
||||
|
||||
## 👀 Watch Mode
|
||||
|
||||
### Auto-run CI on File Changes
|
||||
```bash
|
||||
# Watch and run pre-commit validation
|
||||
make ci-watch
|
||||
|
||||
# Watch and run quick CI
|
||||
make ci-watch-quick
|
||||
|
||||
# Direct script usage
|
||||
./scripts/ci-watch.sh precommit # Fast validation
|
||||
./scripts/ci-watch.sh quick # More thorough
|
||||
```
|
||||
|
||||
### Watch Mode Setup
|
||||
```bash
|
||||
# Install file watcher (one-time setup)
|
||||
sudo apt install inotify-tools
|
||||
|
||||
# Start watching (press Ctrl+C to stop)
|
||||
make ci-watch
|
||||
```
|
||||
|
||||
## 📊 Pipeline Comparison
|
||||
|
||||
| Pipeline | Time | Build | Test | Lint | Security | Math | Docker | Use Case |
|
||||
|----------|------|-------|------|------|----------|------|--------|----------|
|
||||
| `ci-precommit` | 10-30s | ✅ | ✅ | Basic | ❌ | ❌ | ❌ | Pre-commit |
|
||||
| `ci-quick` | 30-60s | ✅ | ✅ | Full | ❌ | ❌ | ❌ | Development |
|
||||
| `ci-dev` | 1-2min | ✅ | ✅ | Full | ✅ | ✅ | ❌ | Daily work |
|
||||
| `ci-full` | 3-5min | ✅ | ✅ | Full | ✅ | ✅ | ✅ | Release |
|
||||
|
||||
## 🎯 Recommended Workflow
|
||||
|
||||
### Daily Development
|
||||
```bash
|
||||
# Morning: Quick validation
|
||||
make ci-quick
|
||||
|
||||
# Before each commit
|
||||
make ci-precommit
|
||||
|
||||
# End of day: Full development check
|
||||
make ci-dev
|
||||
```
|
||||
|
||||
### Release Preparation
|
||||
```bash
|
||||
# Complete validation
|
||||
make ci-full
|
||||
|
||||
# Check all reports
|
||||
ls -la harness/reports/
|
||||
cat harness/reports/pipeline-report.md
|
||||
```
|
||||
|
||||
### Continuous Development
|
||||
```bash
|
||||
# Set up watch mode in terminal
|
||||
make ci-watch
|
||||
|
||||
# Code in another terminal - CI runs automatically on save
|
||||
```
|
||||
|
||||
## 📁 Output and Reports
|
||||
|
||||
### Where Reports Are Stored
|
||||
```bash
|
||||
harness/
|
||||
├── logs/ # Detailed execution logs
|
||||
│ ├── go-deps.log
|
||||
│ ├── unit-tests.log
|
||||
│ ├── golangci-lint.log
|
||||
│ └── docker-build.log
|
||||
└── reports/ # Analysis reports
|
||||
├── pipeline-report.md # Executive summary
|
||||
├── coverage.html # Interactive test coverage
|
||||
├── coverage.out # Coverage data
|
||||
├── math-audit.md # Mathematical validation
|
||||
└── gosec-results.sarif # Security findings
|
||||
```
|
||||
|
||||
### Viewing Reports
|
||||
```bash
|
||||
# Quick summary
|
||||
cat harness/reports/pipeline-report.md
|
||||
|
||||
# Test coverage in browser
|
||||
open harness/reports/coverage.html # macOS
|
||||
xdg-open harness/reports/coverage.html # Linux
|
||||
|
||||
# Coverage percentage
|
||||
grep "total:" harness/reports/coverage.out
|
||||
```
|
||||
|
||||
## 🔧 Customization
|
||||
|
||||
### Environment Variables
|
||||
```bash
|
||||
# Skip components for speed
|
||||
export HARNESS_SKIP_DOCKER=true
|
||||
export HARNESS_SKIP_MATH_AUDIT=true
|
||||
export HARNESS_SKIP_SECURITY=true
|
||||
|
||||
# Performance tuning
|
||||
export HARNESS_PARALLEL_JOBS=8
|
||||
export HARNESS_RUNTIME=podman
|
||||
|
||||
# Custom paths
|
||||
export HARNESS_LOG_DIR=/tmp/ci-logs
|
||||
export HARNESS_REPORT_DIR=/tmp/ci-reports
|
||||
```
|
||||
|
||||
### Creating Custom Pipelines
|
||||
```bash
|
||||
# Custom script example
|
||||
#!/bin/bash
|
||||
# Save as scripts/ci-custom.sh
|
||||
|
||||
HARNESS_SKIP_DOCKER=true \
|
||||
HARNESS_SKIP_SECURITY=true \
|
||||
HARNESS_PARALLEL_JOBS=2 \
|
||||
./harness/local-ci-pipeline.sh
|
||||
```
|
||||
|
||||
## 🚨 Troubleshooting
|
||||
|
||||
### Common Issues and Solutions
|
||||
|
||||
#### Pipeline Fails
|
||||
```bash
|
||||
# Check specific logs
|
||||
cat harness/logs/[failed-step].log
|
||||
|
||||
# Clean and retry
|
||||
make clean
|
||||
go clean -cache -modcache -testcache
|
||||
make ci-quick
|
||||
```
|
||||
|
||||
#### Permission Errors
|
||||
```bash
|
||||
# Make scripts executable
|
||||
chmod +x scripts/ci-*.sh
|
||||
chmod +x harness/local-ci-pipeline.sh
|
||||
```
|
||||
|
||||
#### Container Issues
|
||||
```bash
|
||||
# Check container runtime
|
||||
podman --version || docker --version
|
||||
|
||||
# Use different runtime
|
||||
HARNESS_RUNTIME=docker make ci-container
|
||||
```
|
||||
|
||||
#### Go Module Issues
|
||||
```bash
|
||||
# Fix module problems
|
||||
go mod tidy
|
||||
go mod verify
|
||||
make ci-precommit
|
||||
```
|
||||
|
||||
#### Missing Dependencies
|
||||
```bash
|
||||
# Install required tools
|
||||
go install github.com/golangci/golangci-lint/cmd/golangci-lint@latest
|
||||
|
||||
# For watch mode
|
||||
sudo apt install inotify-tools
|
||||
```
|
||||
|
||||
## 💡 Best Practices
|
||||
|
||||
### Pre-commit Hook Setup
|
||||
```bash
|
||||
# Create git pre-commit hook
|
||||
cat > .git/hooks/pre-commit << 'EOF'
|
||||
#!/bin/bash
|
||||
make ci-precommit
|
||||
EOF
|
||||
|
||||
chmod +x .git/hooks/pre-commit
|
||||
```
|
||||
|
||||
### IDE Integration
|
||||
|
||||
#### VS Code Tasks
|
||||
```json
|
||||
// .vscode/tasks.json
|
||||
{
|
||||
"tasks": [
|
||||
{
|
||||
"label": "CI: Quick",
|
||||
"type": "shell",
|
||||
"command": "make ci-quick",
|
||||
"group": "build"
|
||||
},
|
||||
{
|
||||
"label": "CI: Pre-commit",
|
||||
"type": "shell",
|
||||
"command": "make ci-precommit",
|
||||
"group": "test"
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
#### Keyboard Shortcuts
|
||||
- **F5**: `make ci-precommit` (quick validation)
|
||||
- **Shift+F5**: `make ci-quick` (full validation)
|
||||
- **Ctrl+Shift+T**: `make ci-watch` (watch mode)
|
||||
|
||||
### Performance Tips
|
||||
```bash
|
||||
# Use persistent caches
|
||||
export HARNESS_GOCACHE=$HOME/.cache/mev-bot-go
|
||||
export HARNESS_GOMODCACHE=$HOME/.cache/mev-bot-modules
|
||||
|
||||
# Parallel execution
|
||||
export HARNESS_PARALLEL_JOBS=$(nproc)
|
||||
|
||||
# Use fastest container runtime
|
||||
export HARNESS_RUNTIME=podman
|
||||
```
|
||||
|
||||
## 🎉 Success Indicators
|
||||
|
||||
### What Success Looks Like
|
||||
```bash
|
||||
✅ Completed go-deps
|
||||
✅ Completed go-verify
|
||||
✅ Completed go-tidy-check
|
||||
✅ Completed go-fmt-check
|
||||
✅ Completed go-vet
|
||||
✅ Completed golangci-lint
|
||||
✅ Completed unit-tests
|
||||
✅ Completed build-binary
|
||||
✅ Completed smoke-test
|
||||
🎉 Pipeline completed successfully in 67s
|
||||
```
|
||||
|
||||
### Key Metrics
|
||||
- **Build time**: <30 seconds
|
||||
- **Test coverage**: >80%
|
||||
- **Binary size**: ~26MB
|
||||
- **Zero critical security issues**
|
||||
- **All code quality checks pass**
|
||||
|
||||
The CI/CD pipeline is designed to be fast, reliable, and provide comprehensive validation while being easy to use in daily development workflows.
|
||||
Reference in New Issue
Block a user