mev-beta/docs/5_development/CI_PIPELINE_USAGE.md

CI/CD Pipeline Usage Guide

🚀 Quick Reference

Make Commands (Recommended)

# Pre-commit validation (fastest - 10-30 seconds)
make ci-precommit

# Quick CI pipeline (30-60 seconds)
make ci-quick

# Development CI pipeline (1-2 minutes)
make ci-dev

# Full CI pipeline (3-5 minutes)
make ci-full

# Container-based CI
make ci-container
make ci-container-quick

# Watch mode (auto-run on file changes)
make ci-watch
make ci-watch-quick

Direct Script Usage

# Individual scripts
./scripts/ci-precommit.sh
./scripts/ci-quick.sh
./scripts/ci-dev.sh
./scripts/ci-full.sh
./scripts/ci-container.sh [quick|dev|full]
./scripts/ci-watch.sh [quick|precommit]

📋 Pipeline Types Explained

1. Pre-commit (make ci-precommit)

Purpose: Fastest validation before committing code Time: 10-30 seconds What it does:

• ✅ Build binary
• ✅ Run unit tests
• ✅ Check code formatting
• ✅ Static analysis (go vet)

# Use before every commit
make ci-precommit

2. Quick CI (make ci-quick)

Purpose: Fast comprehensive validation Time: 30-60 seconds What it does:

• ✅ All pre-commit checks
• ✅ Advanced linting
• ❌ Skips: Docker, Math Audit, Security scans

# Good for regular development
make ci-quick

3. Development CI (make ci-dev)

Purpose: Balanced speed and coverage Time: 1-2 minutes What it does:

• ✅ All quick CI checks
• ✅ Math audit validation
• ✅ Security scanning
• ❌ Skips: Docker build only

# Best for daily development workflow
make ci-dev

4. Full CI (make ci-full)

Purpose: Complete validation for releases Time: 3-5 minutes What it does:

• ✅ Everything enabled
• ✅ Docker build and validation
• ✅ Complete security audit
• ✅ Mathematical validation
• ✅ All reports generated

# Use before releases or PR creation
make ci-full

🐳 Container-based CI

When to Use Container CI

• Isolation from host environment
• Consistent build environment
• Testing without local Go installation
• CI/CD server deployment

Container Commands

# Development CI in container
make ci-container

# Quick validation in container
make ci-container-quick

# Direct script with options
./scripts/ci-container.sh quick   # Fast
./scripts/ci-container.sh dev     # Balanced
./scripts/ci-container.sh full    # Complete (no Docker)

Container Requirements

# Ensure container runtime is available
podman --version  # or docker --version

# Install if needed
sudo apt install podman  # or docker.io

👀 Watch Mode

Auto-run CI on File Changes

# Watch and run pre-commit validation
make ci-watch

# Watch and run quick CI
make ci-watch-quick

# Direct script usage
./scripts/ci-watch.sh precommit  # Fast validation
./scripts/ci-watch.sh quick      # More thorough

Watch Mode Setup

# Install file watcher (one-time setup)
sudo apt install inotify-tools

# Start watching (press Ctrl+C to stop)
make ci-watch

📊 Pipeline Comparison

Pipeline	Time	Build	Test	Lint	Security	Math	Docker	Use Case
ci-precommit	10-30s	✅	✅	Basic	❌	❌	❌	Pre-commit
ci-quick	30-60s	✅	✅	Full	❌	❌	❌	Development
ci-dev	1-2min	✅	✅	Full	✅	✅	❌	Daily work
ci-full	3-5min	✅	✅	Full	✅	✅	✅	Release

🎯 Recommended Workflow

Daily Development

# Morning: Quick validation
make ci-quick

# Before each commit
make ci-precommit

# End of day: Full development check
make ci-dev

Release Preparation

# Complete validation
make ci-full

# Check all reports
ls -la harness/reports/
cat harness/reports/pipeline-report.md

Continuous Development

# Set up watch mode in terminal
make ci-watch

# Code in another terminal - CI runs automatically on save

📁 Output and Reports

Where Reports Are Stored

harness/
├── logs/                    # Detailed execution logs
│   ├── go-deps.log
│   ├── unit-tests.log
│   ├── golangci-lint.log
│   └── docker-build.log
└── reports/                 # Analysis reports
    ├── pipeline-report.md   # Executive summary
    ├── coverage.html        # Interactive test coverage
    ├── coverage.out         # Coverage data
    ├── math-audit.md        # Mathematical validation
    └── gosec-results.sarif  # Security findings

Viewing Reports

# Quick summary
cat harness/reports/pipeline-report.md

# Test coverage in browser
open harness/reports/coverage.html  # macOS
xdg-open harness/reports/coverage.html  # Linux

# Coverage percentage
grep "total:" harness/reports/coverage.out

🔧 Customization

Environment Variables

# Skip components for speed
export HARNESS_SKIP_DOCKER=true
export HARNESS_SKIP_MATH_AUDIT=true
export HARNESS_SKIP_SECURITY=true

# Performance tuning
export HARNESS_PARALLEL_JOBS=8
export HARNESS_RUNTIME=podman

# Custom paths
export HARNESS_LOG_DIR=/tmp/ci-logs
export HARNESS_REPORT_DIR=/tmp/ci-reports

Creating Custom Pipelines

# Custom script example
#!/bin/bash
# Save as scripts/ci-custom.sh

HARNESS_SKIP_DOCKER=true \
HARNESS_SKIP_SECURITY=true \
HARNESS_PARALLEL_JOBS=2 \
./harness/local-ci-pipeline.sh

🚨 Troubleshooting

Common Issues and Solutions

Pipeline Fails

# Check specific logs
cat harness/logs/[failed-step].log

# Clean and retry
make clean
go clean -cache -modcache -testcache
make ci-quick

Permission Errors

# Make scripts executable
chmod +x scripts/ci-*.sh
chmod +x harness/local-ci-pipeline.sh

Container Issues

# Check container runtime
podman --version || docker --version

# Use different runtime
HARNESS_RUNTIME=docker make ci-container

Go Module Issues

# Fix module problems
go mod tidy
go mod verify
make ci-precommit

Missing Dependencies

# Install required tools
go install github.com/golangci/golangci-lint/cmd/golangci-lint@latest

# For watch mode
sudo apt install inotify-tools

💡 Best Practices

Pre-commit Hook Setup

# Create git pre-commit hook
cat > .git/hooks/pre-commit << 'EOF'
#!/bin/bash
make ci-precommit
EOF

chmod +x .git/hooks/pre-commit

IDE Integration

VS Code Tasks

// .vscode/tasks.json
{
  "tasks": [
    {
      "label": "CI: Quick",
      "type": "shell",
      "command": "make ci-quick",
      "group": "build"
    },
    {
      "label": "CI: Pre-commit",
      "type": "shell",
      "command": "make ci-precommit",
      "group": "test"
    }
  ]
}

Keyboard Shortcuts

• F5: make ci-precommit (quick validation)
• Shift+F5: make ci-quick (full validation)
• Ctrl+Shift+T: make ci-watch (watch mode)

Performance Tips

# Use persistent caches
export HARNESS_GOCACHE=$HOME/.cache/mev-bot-go
export HARNESS_GOMODCACHE=$HOME/.cache/mev-bot-modules

# Parallel execution
export HARNESS_PARALLEL_JOBS=$(nproc)

# Use fastest container runtime
export HARNESS_RUNTIME=podman

🎉 Success Indicators

What Success Looks Like

✅ Completed go-deps
✅ Completed go-verify
✅ Completed go-tidy-check
✅ Completed go-fmt-check
✅ Completed go-vet
✅ Completed golangci-lint
✅ Completed unit-tests
✅ Completed build-binary
✅ Completed smoke-test
🎉 Pipeline completed successfully in 67s

Key Metrics

• Build time: <30 seconds
• Test coverage: >80%
• Binary size: ~26MB
• Zero critical security issues
• All code quality checks pass

The CI/CD pipeline is designed to be fast, reliable, and provide comprehensive validation while being easy to use in daily development workflows.