ci: add comprehensive CI/CD pipeline with 100% coverage enforcement

Created complete CI/CD infrastructure for V2 development: GitHub Actions Pipeline (.github/workflows/v2-ci.yml): - Pre-flight checks (branch naming, commit messages) - Build & dependency validation - Code quality with 40+ linters (golangci-lint) - Unit tests with MANDATORY 100% coverage enforcement - Integration tests with timeout management - Performance benchmarks (parser < 5ms, detection < 10ms, e2e < 50ms) - Decimal precision validation - Modularity validation (component independence) - Final validation summary with PR comments Code Quality (.golangci.yml): - 40+ enabled linters for comprehensive checks - Cyclomatic complexity limits (max 15) - Magic number detection - Security scanning (gosec) - Style checking with MEV/DEX terminology - Test file exclusions for appropriate linters Build Automation (Makefile): - build, test, test-coverage with 100% enforcement - lint, fmt, vet, security targets - deps-download, deps-verify, deps-tidy, deps-check - validate (full CI/CD locally) - bench (performance benchmarks) - check-modularity, check-circular - Color-coded output for better UX Git Optimization (.gitattributes): - LF normalization for cross-platform consistency - Binary file handling - Diff settings for Go files - Merge strategies - Export-ignore for archives Git Hooks (.git-hooks/): - pre-commit: format, tests, vet, secret detection, go.mod tidy - commit-msg: message format validation - README with installation instructions - install-git-hooks.sh script for easy setup Documentation (docs/planning/05_CI_CD_SETUP.md): - Complete pipeline architecture diagram - Local development workflow - GitHub Actions job descriptions - Performance optimizations (caching, parallel execution) - Failure handling and debugging - Branch protection rules - Deployment process - Best practices and troubleshooting Performance Targets: - Pipeline duration: < 15 minutes - Test coverage: 100% (enforced, non-negotiable) - Parser latency: < 5ms - Arbitrage detection: < 10ms - End-to-end: < 50ms 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
2025-11-10 14:40:08 +01:00
parent 9a92f43edf
commit 24b4d90e98
9 changed files with 2209 additions and 0 deletions
--- a/.git-hooks/README.md
+++ b/.git-hooks/README.md
@@ -0,0 +1,153 @@
+# Git Hooks for MEV Bot V2
+
+This directory contains Git hooks to ensure code quality and consistency.
+
+## Installation
+
+Run these commands from the repository root:
+
+```bash
+# Make hooks executable
+chmod +x .git-hooks/*
+
+# Install pre-commit hook
+ln -sf ../../.git-hooks/pre-commit .git/hooks/pre-commit
+
+# Install commit-msg hook
+ln -sf ../../.git-hooks/commit-msg .git/hooks/commit-msg
+```
+
+Or use the provided installation script:
+
+```bash
+./scripts/install-git-hooks.sh
+```
+
+## Available Hooks
+
+### pre-commit
+
+Runs before each commit and performs:
+
+1. **Branch Name Validation** - Ensures correct naming convention
+2. **Merge Conflict Detection** - Prevents committing conflict markers
+3. **Secret Detection** - Scans for passwords, API keys, tokens
+4. **Dependency Management** - Auto-tidies go.mod and go.sum
+5. **Code Formatting** - Auto-formats Go code with gofmt
+6. **Quick Tests** - Runs tests on changed packages
+7. **Go Vet** - Runs static analysis
+8. **File Size Check** - Warns about large files
+
+### commit-msg
+
+Validates commit message format:
+
+**Required Format:**
+```
+type(scope): description
+
+Optional body explaining the change
+
+🤖 Generated with [Claude Code](https://claude.com/claude-code)
+Co-Authored-By: Claude <noreply@anthropic.com>
+```
+
+**Valid Types:**
+- `feat` - New feature
+- `fix` - Bug fix
+- `perf` - Performance improvement
+- `refactor` - Code refactoring
+- `test` - Tests
+- `docs` - Documentation
+- `build` - Build system
+- `ci` - CI/CD
+
+**Example:**
+```
+feat(parsers): add UniswapV2 parser with event validation
+
+- Implements ParseLog() for Swap events
+- Adds token extraction from pool cache
+- Includes comprehensive validation rules
+- Achieves 100% test coverage
+
+🤖 Generated with [Claude Code](https://claude.com/claude-code)
+Co-Authored-By: Claude <noreply@anthropic.com>
+```
+
+## Bypassing Hooks (Emergency Only)
+
+If you absolutely must bypass hooks:
+
+```bash
+git commit --no-verify -m "emergency fix"
+```
+
+**⚠️ Warning:** Only use in emergencies. CI/CD will still enforce all checks.
+
+## Troubleshooting
+
+### Hook not executing
+
+1. Check if hook is executable:
+   ```bash
+   ls -l .git/hooks/pre-commit
+   ```
+
+2. If not, make it executable:
+   ```bash
+   chmod +x .git/hooks/pre-commit
+   ```
+
+### Hook failing unexpectedly
+
+1. Run the hook manually to see errors:
+   ```bash
+   .git/hooks/pre-commit
+   ```
+
+2. Check that all required tools are installed:
+   ```bash
+   which gofmt
+   which go
+   ```
+
+### Disabling hooks temporarily
+
+```bash
+# Disable all hooks
+git config core.hooksPath /dev/null
+
+# Re-enable hooks
+git config --unset core.hooksPath
+```
+
+## Best Practices
+
+1. **Never bypass hooks** unless absolutely necessary
+2. **Fix issues** instead of bypassing
+3. **Keep hooks fast** - they run on every commit
+4. **Test hooks locally** before committing to shared repository
+5. **Document any new hooks** added to this directory
+
+## Performance
+
+Hooks are designed to be fast:
+
+- **Pre-commit**: Typically < 5 seconds
+- **Commit-msg**: < 1 second
+
+If hooks are slow, consider:
+
+1. Only testing changed packages (already implemented)
+2. Using `--short` flag for tests (already implemented)
+3. Running full tests in CI/CD instead
+
+## Maintenance
+
+Review and update hooks periodically:
+
+1. Add new checks as project evolves
+2. Remove obsolete checks
+3. Optimize performance
+4. Keep documentation up to date
--- a/.git-hooks/commit-msg
+++ b/.git-hooks/commit-msg
@@ -0,0 +1,90 @@
+#!/bin/bash
+#
+# Commit message hook for MEV Bot V2
+# Validates commit message format
+#
+# Install: ln -sf ../../.git-hooks/commit-msg .git/hooks/commit-msg
+#
+
+# Colors
+RED='\033[0;31m'
+GREEN='\033[0;32m'
+YELLOW='\033[1;33m'
+NC='\033[0m' # No Color
+
+COMMIT_MSG_FILE=$1
+COMMIT_MSG=$(cat "$COMMIT_MSG_FILE")
+
+# Skip if this is a merge commit
+if git rev-parse -q --verify MERGE_HEAD > /dev/null; then
+    echo -e "${GREEN}ℹ️  Merge commit detected, skipping validation${NC}"
+    exit 0
+fi
+
+# Skip if this is an amend
+if [ -n "$GIT_EDITOR" ]; then
+    echo -e "${GREEN}ℹ️  Amend detected, skipping validation${NC}"
+    exit 0
+fi
+
+echo -e "${GREEN}📝 Validating commit message...${NC}"
+
+# Required format: type(scope): description
+# type: feat, fix, perf, refactor, test, docs, build, ci
+# scope: component name (parsers, cache, validation, etc.)
+
+PATTERN="^(feat|fix|perf|refactor|test|docs|build|ci)\([a-z0-9-]+\): .{10,}"
+
+if ! echo "$COMMIT_MSG" | grep -qE "$PATTERN"; then
+    echo -e "${RED}❌ Invalid commit message format${NC}"
+    echo ""
+    echo -e "${YELLOW}Required format:${NC}"
+    echo -e "  type(scope): brief description"
+    echo ""
+    echo -e "${YELLOW}Valid types:${NC}"
+    echo -e "  feat     - New feature"
+    echo -e "  fix      - Bug fix"
+    echo -e "  perf     - Performance improvement"
+    echo -e "  refactor - Code refactoring"
+    echo -e "  test     - Adding or updating tests"
+    echo -e "  docs     - Documentation updates"
+    echo -e "  build    - Build system changes"
+    echo -e "  ci       - CI/CD changes"
+    echo ""
+    echo -e "${YELLOW}Example:${NC}"
+    echo -e "  feat(parsers): add UniswapV2 parser with event validation"
+    echo ""
+    echo -e "${YELLOW}Your message:${NC}"
+    echo -e "  $COMMIT_MSG"
+    echo ""
+    exit 1
+fi
+
+# Check for minimum description length
+DESCRIPTION=$(echo "$COMMIT_MSG" | head -n1 | sed 's/^[^:]*: //')
+if [ ${#DESCRIPTION} -lt 10 ]; then
+    echo -e "${RED}❌ Commit description too short (minimum 10 characters)${NC}"
+    echo -e "${YELLOW}Your description: $DESCRIPTION (${#DESCRIPTION} chars)${NC}"
+    exit 1
+fi
+
+# Check for maximum line length (72 chars for first line)
+FIRST_LINE=$(echo "$COMMIT_MSG" | head -n1)
+if [ ${#FIRST_LINE} -gt 72 ]; then
+    echo -e "${YELLOW}⚠️  Warning: First line exceeds 72 characters (${#FIRST_LINE} chars)${NC}"
+    echo -e "${YELLOW}   Consider shortening the description${NC}"
+    echo ""
+fi
+
+# Encourage including coverage info for test changes
+if echo "$COMMIT_MSG" | grep -q "^test"; then
+    if ! echo "$COMMIT_MSG" | grep -qi "coverage"; then
+        echo -e "${YELLOW}💡 Tip: Consider including coverage info in test commits${NC}"
+        echo ""
+    fi
+fi
+
+echo -e "${GREEN}✅ Commit message format valid${NC}"
+echo ""
+
+exit 0
--- a/.git-hooks/pre-commit
+++ b/.git-hooks/pre-commit
@@ -0,0 +1,202 @@
+#!/bin/bash
+#
+# Pre-commit hook for MEV Bot V2
+# Ensures code quality and consistency before commits
+#
+# Install: ln -sf ../../.git-hooks/pre-commit .git/hooks/pre-commit
+#
+
+set -e
+
+# Colors
+RED='\033[0;31m'
+GREEN='\033[0;32m'
+YELLOW='\033[1;33m'
+NC='\033[0m' # No Color
+
+echo -e "${GREEN}🔍 Running pre-commit checks...${NC}"
+echo ""
+
+# ==============================================================================
+# 1. CHECK BRANCH NAME
+# ==============================================================================
+BRANCH_NAME=$(git branch --show-current)
+
+if [[ "$BRANCH_NAME" == "feature/v2-prep" ]] || [[ "$BRANCH_NAME" == "master" ]]; then
+    echo -e "${YELLOW}⚠️  Warning: Committing to protected branch: $BRANCH_NAME${NC}"
+    echo -e "${YELLOW}   Consider creating a feature branch instead${NC}"
+    echo ""
+fi
+
+if [[ "$BRANCH_NAME" =~ ^feature/v2/ ]]; then
+    # Validate branch naming convention
+    if [[ ! "$BRANCH_NAME" =~ ^feature/v2/[a-z0-9-]+/[A-Z0-9]+-[0-9]+-[a-z0-9-]+$ ]]; then
+        echo -e "${YELLOW}⚠️  Branch name doesn't follow convention:${NC}"
+        echo -e "${YELLOW}   feature/v2/<component>/<TASK-ID>-<description>${NC}"
+        echo ""
+    fi
+fi
+
+# ==============================================================================
+# 2. CHECK FOR MERGE CONFLICTS
+# ==============================================================================
+echo -e "${GREEN}📋 Checking for merge conflicts...${NC}"
+if git diff --cached --name-only | xargs grep -l "^<<<<<<< HEAD" 2>/dev/null; then
+    echo -e "${RED}❌ Merge conflict markers found${NC}"
+    echo -e "${RED}   Resolve conflicts before committing${NC}"
+    exit 1
+fi
+echo -e "${GREEN}✅ No merge conflicts${NC}"
+echo ""
+
+# ==============================================================================
+# 3. CHECK FOR FORBIDDEN PATTERNS
+# ==============================================================================
+echo -e "${GREEN}🔒 Checking for secrets and forbidden patterns...${NC}"
+
+# Check for common secret patterns
+if git diff --cached --name-only -z | xargs -0 grep -E "password|secret|api[_-]?key|private[_-]?key|token" --include="*.go" 2>/dev/null | grep -v "test" | grep -v "example"; then
+    echo -e "${RED}❌ Potential secrets found${NC}"
+    echo -e "${RED}   Remove secrets before committing${NC}"
+    exit 1
+fi
+
+# Check for debugging statements
+if git diff --cached --name-only -z | xargs -0 grep -E "fmt\.Println|log\.Println|panic\(|TODO.*URGENT|FIXME.*CRITICAL" --include="*.go" 2>/dev/null; then
+    echo -e "${YELLOW}⚠️  Warning: Found debugging statements or urgent TODOs${NC}"
+    echo -e "${YELLOW}   Consider removing or creating issues for them${NC}"
+    echo ""
+fi
+
+echo -e "${GREEN}✅ No forbidden patterns found${NC}"
+echo ""
+
+# ==============================================================================
+# 4. GO MOD TIDY CHECK
+# ==============================================================================
+if [ -f "go.mod" ]; then
+    echo -e "${GREEN}📦 Checking if go.mod is tidy...${NC}"
+
+    # Save current go.mod and go.sum
+    cp go.mod go.mod.backup
+    cp go.sum go.sum.backup
+
+    # Run go mod tidy
+    go mod tidy
+
+    # Check if anything changed
+    if ! diff -q go.mod go.mod.backup > /dev/null 2>&1 || ! diff -q go.sum go.sum.backup > /dev/null 2>&1; then
+        echo -e "${YELLOW}⚠️  go.mod or go.sum was not tidy${NC}"
+        echo -e "${YELLOW}   Auto-fixed and staged${NC}"
+        git add go.mod go.sum
+    fi
+
+    # Clean up backups
+    rm -f go.mod.backup go.sum.backup
+
+    echo -e "${GREEN}✅ Dependencies are tidy${NC}"
+    echo ""
+fi
+
+# ==============================================================================
+# 5. CODE FORMATTING
+# ==============================================================================
+echo -e "${GREEN}🎨 Checking code formatting...${NC}"
+
+# Get list of staged Go files
+STAGED_GO_FILES=$(git diff --cached --name-only --diff-filter=ACM | grep "\.go$" || true)
+
+if [ -n "$STAGED_GO_FILES" ]; then
+    # Check formatting
+    UNFORMATTED=$(gofmt -l $STAGED_GO_FILES 2>/dev/null || true)
+
+    if [ -n "$UNFORMATTED" ]; then
+        echo -e "${YELLOW}⚠️  Auto-formatting files:${NC}"
+        echo "$UNFORMATTED"
+
+        # Auto-format files
+        echo "$UNFORMATTED" | xargs gofmt -w -s
+
+        # Re-stage formatted files
+        echo "$UNFORMATTED" | xargs git add
+
+        echo -e "${GREEN}✅ Code formatted and re-staged${NC}"
+    else
+        echo -e "${GREEN}✅ All files properly formatted${NC}"
+    fi
+else
+    echo -e "${GREEN}ℹ️  No Go files to format${NC}"
+fi
+echo ""
+
+# ==============================================================================
+# 6. RUN TESTS ON CHANGED FILES
+# ==============================================================================
+if [ -n "$STAGED_GO_FILES" ]; then
+    echo -e "${GREEN}🧪 Running tests on changed packages...${NC}"
+
+    # Get unique package directories
+    PACKAGES=$(echo "$STAGED_GO_FILES" | xargs -n1 dirname | sort -u | sed 's/$/\/.../')
+
+    # Run tests with timeout
+    if ! go test -short -timeout=2m $PACKAGES 2>&1; then
+        echo -e "${RED}❌ Tests failed${NC}"
+        echo -e "${RED}   Fix tests before committing${NC}"
+        exit 1
+    fi
+
+    echo -e "${GREEN}✅ Tests passed${NC}"
+else
+    echo -e "${GREEN}ℹ️  No Go files changed, skipping tests${NC}"
+fi
+echo ""
+
+# ==============================================================================
+# 7. RUN GO VET
+# ==============================================================================
+if [ -n "$STAGED_GO_FILES" ]; then
+    echo -e "${GREEN}🔍 Running go vet...${NC}"
+
+    if ! go vet ./... 2>&1; then
+        echo -e "${RED}❌ go vet found issues${NC}"
+        echo -e "${RED}   Fix issues before committing${NC}"
+        exit 1
+    fi
+
+    echo -e "${GREEN}✅ go vet passed${NC}"
+fi
+echo ""
+
+# ==============================================================================
+# 8. CHECK FILE SIZES
+# ==============================================================================
+echo -e "${GREEN}📏 Checking file sizes...${NC}"
+
+LARGE_FILES=$(git diff --cached --name-only | xargs -I {} sh -c 'if [ -f "{}" ]; then stat -f%z "{}" 2>/dev/null || stat -c%s "{}" 2>/dev/null; fi' | awk '$1 > 1048576 {print}' || true)
+
+if [ -n "$LARGE_FILES" ]; then
+    echo -e "${YELLOW}⚠️  Warning: Large files detected (>1MB)${NC}"
+    echo -e "${YELLOW}   Consider if these should be committed${NC}"
+    git diff --cached --name-only | while read file; do
+        size=$(stat -f%z "$file" 2>/dev/null || stat -c%s "$file" 2>/dev/null)
+        if [ "$size" -gt 1048576 ]; then
+            size_mb=$(echo "scale=2; $size / 1048576" | bc)
+            echo -e "${YELLOW}   $file: ${size_mb}MB${NC}"
+        fi
+    done
+    echo ""
+fi
+
+echo -e "${GREEN}✅ File size check complete${NC}"
+echo ""
+
+# ==============================================================================
+# 9. FINAL SUMMARY
+# ==============================================================================
+echo -e "${GREEN}╔══════════════════════════════════════╗${NC}"
+echo -e "${GREEN}║  ✅ PRE-COMMIT CHECKS PASSED ✅      ║${NC}"
+echo -e "${GREEN}╚══════════════════════════════════════╝${NC}"
+echo ""
+echo -e "${GREEN}Proceeding with commit...${NC}"
+
+exit 0