mev-beta/docs/5_development/CI_PIPELINE_USAGE.md

# CI/CD Pipeline Usage Guide

## 🚀 Quick Reference

### Make Commands (Recommended)
```bash
# Pre-commit validation (fastest - 10-30 seconds)
make ci-precommit

# Quick CI pipeline (30-60 seconds)
make ci-quick

# Development CI pipeline (1-2 minutes)
make ci-dev

# Full CI pipeline (3-5 minutes)
make ci-full

# Container-based CI
make ci-container
make ci-container-quick

# Watch mode (auto-run on file changes)
make ci-watch
make ci-watch-quick
```

### Direct Script Usage
```bash
# Individual scripts
./scripts/ci-precommit.sh
./scripts/ci-quick.sh
./scripts/ci-dev.sh
./scripts/ci-full.sh
./scripts/ci-container.sh [quick|dev|full]
./scripts/ci-watch.sh [quick|precommit]
```

## 📋 Pipeline Types Explained

### 1. Pre-commit (`make ci-precommit`)
**Purpose**: Fastest validation before committing code
**Time**: 10-30 seconds
**What it does**:
- ✅ Build binary
- ✅ Run unit tests
- ✅ Check code formatting
- ✅ Static analysis (go vet)

```bash
# Use before every commit
make ci-precommit
```

### 2. Quick CI (`make ci-quick`)
**Purpose**: Fast comprehensive validation
**Time**: 30-60 seconds
**What it does**:
- ✅ All pre-commit checks
- ✅ Advanced linting
- ❌ Skips: Docker, Math Audit, Security scans

```bash
# Good for regular development
make ci-quick
```

### 3. Development CI (`make ci-dev`)
**Purpose**: Balanced speed and coverage
**Time**: 1-2 minutes
**What it does**:
- ✅ All quick CI checks
- ✅ Math audit validation
- ✅ Security scanning
- ❌ Skips: Docker build only

```bash
# Best for daily development workflow
make ci-dev
```

### 4. Full CI (`make ci-full`)
**Purpose**: Complete validation for releases
**Time**: 3-5 minutes
**What it does**:
- ✅ Everything enabled
- ✅ Docker build and validation
- ✅ Complete security audit
- ✅ Mathematical validation
- ✅ All reports generated

```bash
# Use before releases or PR creation
make ci-full
```

## 🐳 Container-based CI

### When to Use Container CI
- Isolation from host environment
- Consistent build environment
- Testing without local Go installation
- CI/CD server deployment

### Container Commands
```bash
# Development CI in container
make ci-container

# Quick validation in container
make ci-container-quick

# Direct script with options
./scripts/ci-container.sh quick   # Fast
./scripts/ci-container.sh dev     # Balanced
./scripts/ci-container.sh full    # Complete (no Docker)
```

### Container Requirements
```bash
# Ensure container runtime is available
podman --version  # or docker --version

# Install if needed
sudo apt install podman  # or docker.io
```

## 👀 Watch Mode

### Auto-run CI on File Changes
```bash
# Watch and run pre-commit validation
make ci-watch

# Watch and run quick CI
make ci-watch-quick

# Direct script usage
./scripts/ci-watch.sh precommit  # Fast validation
./scripts/ci-watch.sh quick      # More thorough
```

### Watch Mode Setup
```bash
# Install file watcher (one-time setup)
sudo apt install inotify-tools

# Start watching (press Ctrl+C to stop)
make ci-watch
```

## 📊 Pipeline Comparison

| Pipeline | Time | Build | Test | Lint | Security | Math | Docker | Use Case |
|----------|------|-------|------|------|----------|------|--------|----------|
| `ci-precommit` | 10-30s | ✅ | ✅ | Basic | ❌ | ❌ | ❌ | Pre-commit |
| `ci-quick` | 30-60s | ✅ | ✅ | Full | ❌ | ❌ | ❌ | Development |
| `ci-dev` | 1-2min | ✅ | ✅ | Full | ✅ | ✅ | ❌ | Daily work |
| `ci-full` | 3-5min | ✅ | ✅ | Full | ✅ | ✅ | ✅ | Release |

## 🎯 Recommended Workflow

### Daily Development
```bash
# Morning: Quick validation
make ci-quick

# Before each commit
make ci-precommit

# End of day: Full development check
make ci-dev
```

### Release Preparation
```bash
# Complete validation
make ci-full

# Check all reports
ls -la harness/reports/
cat harness/reports/pipeline-report.md
```

### Continuous Development
```bash
# Set up watch mode in terminal
make ci-watch

# Code in another terminal - CI runs automatically on save
```

## 📁 Output and Reports

### Where Reports Are Stored
```bash
harness/
├── logs/                    # Detailed execution logs
│   ├── go-deps.log
│   ├── unit-tests.log
│   ├── golangci-lint.log
│   └── docker-build.log
└── reports/                 # Analysis reports
    ├── pipeline-report.md   # Executive summary
    ├── coverage.html        # Interactive test coverage
    ├── coverage.out         # Coverage data
    ├── math-audit.md        # Mathematical validation
    └── gosec-results.sarif  # Security findings
```

### Viewing Reports
```bash
# Quick summary
cat harness/reports/pipeline-report.md

# Test coverage in browser
open harness/reports/coverage.html  # macOS
xdg-open harness/reports/coverage.html  # Linux

# Coverage percentage
grep "total:" harness/reports/coverage.out
```

## 🔧 Customization

### Environment Variables
```bash
# Skip components for speed
export HARNESS_SKIP_DOCKER=true
export HARNESS_SKIP_MATH_AUDIT=true
export HARNESS_SKIP_SECURITY=true

# Performance tuning
export HARNESS_PARALLEL_JOBS=8
export HARNESS_RUNTIME=podman

# Custom paths
export HARNESS_LOG_DIR=/tmp/ci-logs
export HARNESS_REPORT_DIR=/tmp/ci-reports
```

### Creating Custom Pipelines
```bash
# Custom script example
#!/bin/bash
# Save as scripts/ci-custom.sh

HARNESS_SKIP_DOCKER=true \
HARNESS_SKIP_SECURITY=true \
HARNESS_PARALLEL_JOBS=2 \
./harness/local-ci-pipeline.sh
```

## 🚨 Troubleshooting

### Common Issues and Solutions

#### Pipeline Fails
```bash
# Check specific logs
cat harness/logs/[failed-step].log

# Clean and retry
make clean
go clean -cache -modcache -testcache
make ci-quick
```

#### Permission Errors
```bash
# Make scripts executable
chmod +x scripts/ci-*.sh
chmod +x harness/local-ci-pipeline.sh
```

#### Container Issues
```bash
# Check container runtime
podman --version || docker --version

# Use different runtime
HARNESS_RUNTIME=docker make ci-container
```

#### Go Module Issues
```bash
# Fix module problems
go mod tidy
go mod verify
make ci-precommit
```

#### Missing Dependencies
```bash
# Install required tools
go install github.com/golangci/golangci-lint/cmd/golangci-lint@latest

# For watch mode
sudo apt install inotify-tools
```

## 💡 Best Practices

### Pre-commit Hook Setup
```bash
# Create git pre-commit hook
cat > .git/hooks/pre-commit << 'EOF'
#!/bin/bash
make ci-precommit
EOF

chmod +x .git/hooks/pre-commit
```

### IDE Integration

#### VS Code Tasks
```json
// .vscode/tasks.json
{
  "tasks": [
    {
      "label": "CI: Quick",
      "type": "shell",
      "command": "make ci-quick",
      "group": "build"
    },
    {
      "label": "CI: Pre-commit",
      "type": "shell",
      "command": "make ci-precommit",
      "group": "test"
    }
  ]
}
```

#### Keyboard Shortcuts
- **F5**: `make ci-precommit` (quick validation)
- **Shift+F5**: `make ci-quick` (full validation)
- **Ctrl+Shift+T**: `make ci-watch` (watch mode)

### Performance Tips
```bash
# Use persistent caches
export HARNESS_GOCACHE=$HOME/.cache/mev-bot-go
export HARNESS_GOMODCACHE=$HOME/.cache/mev-bot-modules

# Parallel execution
export HARNESS_PARALLEL_JOBS=$(nproc)

# Use fastest container runtime
export HARNESS_RUNTIME=podman
```

## 🎉 Success Indicators

### What Success Looks Like
```bash
✅ Completed go-deps
✅ Completed go-verify
✅ Completed go-tidy-check
✅ Completed go-fmt-check
✅ Completed go-vet
✅ Completed golangci-lint
✅ Completed unit-tests
✅ Completed build-binary
✅ Completed smoke-test
🎉 Pipeline completed successfully in 67s
```

### Key Metrics
- **Build time**: <30 seconds
- **Test coverage**: >80%
- **Binary size**: ~26MB
- **Zero critical security issues**
- **All code quality checks pass**

The CI/CD pipeline is designed to be fast, reliable, and provide comprehensive validation while being easy to use in daily development workflows.