feat(prod): complete production deployment with Podman containerization
- Migrate from Docker to Podman for enhanced security (rootless containers) - Add production-ready Dockerfile with multi-stage builds - Configure production environment with Arbitrum mainnet RPC endpoints - Add comprehensive test coverage for core modules (exchanges, execution, profitability) - Implement production audit and deployment documentation - Update deployment scripts for production environment - Add container runtime and health monitoring scripts - Document RPC limitations and remediation strategies - Implement token metadata caching and pool validation This commit prepares the MEV bot for production deployment on Arbitrum with full containerization, security hardening, and operational tooling. 🤖 Generated with Claude Code Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
Krypto Kajun
514
docs/PODMAN_SETUP.md
Normal file
514
docs/PODMAN_SETUP.md
Normal file
@@ -0,0 +1,514 @@
|
||||
# Podman-First Development Setup
|
||||
|
||||
**Date:** November 6, 2025
|
||||
**Status:** ✅ Complete - Podman-first with Docker fallback
|
||||
**Priority:** All development and production workflows now support Podman
|
||||
|
||||
---
|
||||
|
||||
## Overview
|
||||
|
||||
The MEV bot project is now **Podman-first** with automatic fallback to Docker if available. All scripts, Makefiles, and container operations now intelligently detect and use the best available container runtime.
|
||||
|
||||
### Why Podman?
|
||||
|
||||
- ✅ **Rootless**: Runs without sudo/root privileges
|
||||
- ✅ **Daemonless**: No background daemon required
|
||||
- ✅ **Compatible**: Docker-compatible CLI
|
||||
- ✅ **Secure**: Better security model than Docker
|
||||
- ✅ **Nested**: Supports Podman-in-Podman for dev workflows
|
||||
|
||||
---
|
||||
|
||||
## Quick Start
|
||||
|
||||
### 1. Install Podman
|
||||
|
||||
```bash
|
||||
# Ubuntu/Debian
|
||||
sudo apt-get update
|
||||
sudo apt-get install -y podman podman-compose
|
||||
|
||||
# Verify installation
|
||||
podman --version
|
||||
podman-compose --version
|
||||
```
|
||||
|
||||
### 2. Test Podman Setup
|
||||
|
||||
```bash
|
||||
# Verify runtime detection
|
||||
./scripts/container-runtime.sh status
|
||||
|
||||
# Output should show:
|
||||
# Runtime: podman
|
||||
# Compose: podman-compose
|
||||
```
|
||||
|
||||
### 3. Run Tests in Podman
|
||||
|
||||
```bash
|
||||
# Development CI
|
||||
podman compose -f docker-compose.test.yml up test-unit
|
||||
|
||||
# Full test suite
|
||||
podman compose -f docker-compose.test.yml up test-coverage
|
||||
|
||||
# Security scan
|
||||
podman compose -f docker-compose.test.yml up test-security
|
||||
|
||||
# All tests
|
||||
podman compose -f docker-compose.test.yml up
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Container Runtime Detection
|
||||
|
||||
### Automatic Detection
|
||||
|
||||
All scripts now use `scripts/container-runtime.sh` for intelligent runtime detection:
|
||||
|
||||
```bash
|
||||
# Usage
|
||||
./scripts/container-runtime.sh status # Show current runtime
|
||||
./scripts/container-runtime.sh runtime # Get runtime name
|
||||
./scripts/container-runtime.sh compose # Get compose command
|
||||
./scripts/container-runtime.sh socket # Get socket path
|
||||
```
|
||||
|
||||
### Priority Order
|
||||
|
||||
1. **Podman** (preferred - rootless, daemonless)
|
||||
2. **Docker** (fallback if Podman not available)
|
||||
|
||||
### Detection Features
|
||||
|
||||
- ✅ Detects if running inside container (rootless mode)
|
||||
- ✅ Finds socket paths automatically
|
||||
- ✅ Determines correct compose command
|
||||
- ✅ Supports DinD (Docker in Docker) and PinP (Podman in Podman)
|
||||
|
||||
---
|
||||
|
||||
## Updated Scripts
|
||||
|
||||
### CI Scripts
|
||||
|
||||
**`scripts/ci-container.sh`** - Run CI in isolated container
|
||||
```bash
|
||||
./scripts/ci-container.sh quick # Fast validation (30-60s)
|
||||
./scripts/ci-container.sh dev # Development pipeline (1-2min)
|
||||
./scripts/ci-container.sh full # Complete validation (2-3min)
|
||||
```
|
||||
|
||||
**Features:**
|
||||
- ✅ Auto-detects Podman or Docker
|
||||
- ✅ Mounts socket for DinD support
|
||||
- ✅ Shares Go cache for performance
|
||||
- ✅ Works in nested containers
|
||||
|
||||
### Deployment Scripts
|
||||
|
||||
**`scripts/deploy-production.sh`** - Production deployment
|
||||
```bash
|
||||
./scripts/deploy-production.sh
|
||||
```
|
||||
|
||||
**Updated to:**
|
||||
- ✅ Use Podman by default
|
||||
- ✅ Fall back to Docker
|
||||
- ✅ Use `podman-compose` or `docker-compose`
|
||||
- ✅ Show correct log commands for active runtime
|
||||
|
||||
---
|
||||
|
||||
## Makefile Commands
|
||||
|
||||
All `make` targets work with both Podman and Docker automatically:
|
||||
|
||||
```bash
|
||||
# Build
|
||||
make build # Build with native toolchain
|
||||
make build-mm # Build market manager
|
||||
|
||||
# Tests
|
||||
make test # Run all tests
|
||||
make test-basic # Quick tests
|
||||
make test-coverage # With coverage report
|
||||
make test-unit # Unit tests only
|
||||
|
||||
# CI/CD (now Podman-aware)
|
||||
make ci-precommit # Fast pre-commit checks
|
||||
make ci-quick # Quick CI pipeline
|
||||
make ci-dev # Development CI
|
||||
make ci-full # Full CI pipeline
|
||||
make ci-container # CI in container (Podman/Docker)
|
||||
|
||||
# Development
|
||||
make dev-setup # Setup dev environment
|
||||
make dev-workflow # Format + Vet + Lint + Tests
|
||||
make debug # Debug mode
|
||||
|
||||
# All existing commands work with Podman/Docker detection
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Docker-Compose Compatibility
|
||||
|
||||
### Test Compose File
|
||||
|
||||
**`docker-compose.test.yml`** - All services now work with both:
|
||||
|
||||
```bash
|
||||
# Using Podman
|
||||
podman compose -f docker-compose.test.yml up test-unit
|
||||
podman compose -f docker-compose.test.yml up test-coverage
|
||||
podman compose -f docker-compose.test.yml up test-security
|
||||
|
||||
# Using Docker (if available)
|
||||
docker compose -f docker-compose.test.yml up test-unit
|
||||
docker-compose -f docker-compose.test.yml up test-unit
|
||||
```
|
||||
|
||||
### Production Compose File
|
||||
|
||||
**`docker-compose.production.yaml`** - Works with either runtime:
|
||||
|
||||
```bash
|
||||
# Using Podman
|
||||
podman compose -f docker-compose.production.yaml up -d
|
||||
|
||||
# Using Docker
|
||||
docker compose -f docker-compose.production.yaml up -d
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Development Workflow with Podman
|
||||
|
||||
### Local Development (No Container)
|
||||
|
||||
```bash
|
||||
# Standard Go development
|
||||
make build
|
||||
make test
|
||||
./bin/mev-bot start
|
||||
```
|
||||
|
||||
### Containerized Development
|
||||
|
||||
```bash
|
||||
# Run in isolated container
|
||||
./scripts/ci-container.sh dev
|
||||
|
||||
# Or with compose
|
||||
podman compose -f docker-compose.test.yml up test-unit
|
||||
```
|
||||
|
||||
### Podman-in-Podman (Nested Containers)
|
||||
|
||||
For dev environments that need to run containers inside containers:
|
||||
|
||||
```bash
|
||||
# Check if running in container
|
||||
./scripts/container-runtime.sh status
|
||||
# Shows: Inside Container: true
|
||||
|
||||
# Socket is automatically mounted
|
||||
# Scripts detect and use the parent's socket
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Common Commands
|
||||
|
||||
### Build & Test
|
||||
|
||||
```bash
|
||||
# Build application
|
||||
podman build -t mev-bot:latest .
|
||||
|
||||
# Run tests
|
||||
podman compose -f docker-compose.test.yml up test-unit
|
||||
|
||||
# Generate coverage
|
||||
podman compose -f docker-compose.test.yml up test-coverage
|
||||
|
||||
# Security scan
|
||||
podman compose -f docker-compose.test.yml up test-security
|
||||
```
|
||||
|
||||
### Container Operations
|
||||
|
||||
```bash
|
||||
# List images
|
||||
podman images | grep mev-bot
|
||||
|
||||
# List containers
|
||||
podman ps -a
|
||||
|
||||
# View logs
|
||||
podman logs <container_id>
|
||||
|
||||
# Remove image
|
||||
podman rmi mev-bot:latest
|
||||
|
||||
# Cleanup (prune unused)
|
||||
podman system prune -a
|
||||
```
|
||||
|
||||
### Debugging
|
||||
|
||||
```bash
|
||||
# Interactive shell in container
|
||||
podman run -it --rm -v $(pwd):/app golang:1.25-alpine sh
|
||||
|
||||
# Build with verbose output
|
||||
podman build --no-cache -t mev-bot:latest .
|
||||
|
||||
# Inspect image
|
||||
podman inspect mev-bot:latest
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### "podman: command not found"
|
||||
|
||||
**Solution:** Install Podman
|
||||
```bash
|
||||
sudo apt-get install -y podman podman-compose
|
||||
```
|
||||
|
||||
### "Cannot connect to socket"
|
||||
|
||||
**Solution:** Check socket permissions
|
||||
```bash
|
||||
# Find socket
|
||||
./scripts/container-runtime.sh socket
|
||||
|
||||
# Verify permissions
|
||||
ls -la /run/user/$UID/podman/podman.sock
|
||||
|
||||
# Restart Podman
|
||||
systemctl --user restart podman
|
||||
```
|
||||
|
||||
### "Failed to pull image"
|
||||
|
||||
**Solution:** Check network/registry
|
||||
```bash
|
||||
# Test pull
|
||||
podman pull golang:1.25-alpine
|
||||
|
||||
# Check registry mirrors
|
||||
podman run --rm golang:1.25-alpine apk update
|
||||
```
|
||||
|
||||
### "No space left on device"
|
||||
|
||||
**Solution:** Cleanup images and containers
|
||||
```bash
|
||||
# Remove unused images
|
||||
podman image prune -a
|
||||
|
||||
# Remove stopped containers
|
||||
podman container prune
|
||||
|
||||
# Full cleanup
|
||||
podman system prune -a --volumes
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Environment Detection
|
||||
|
||||
### Auto-Detection in Scripts
|
||||
|
||||
All scripts now automatically detect the runtime:
|
||||
|
||||
```bash
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
# Load container runtime detection
|
||||
source "$(dirname "$0")/container-runtime.sh" init
|
||||
|
||||
# Now available:
|
||||
# - $CONTAINER_RUNTIME: "podman" or "docker"
|
||||
# - $COMPOSE_CMD: Full compose command (podman-compose, docker-compose, etc)
|
||||
# - $CONTAINER_SOCKET: Socket path for DinD
|
||||
# - $INSIDE_CONTAINER: "true" or "false"
|
||||
```
|
||||
|
||||
### Usage in Custom Scripts
|
||||
|
||||
```bash
|
||||
# Source the helper
|
||||
source ./scripts/container-runtime.sh init
|
||||
|
||||
# Use the detected runtime
|
||||
$CONTAINER_RUNTIME run --rm alpine:latest echo "Hello"
|
||||
|
||||
# Use compose
|
||||
$COMPOSE_CMD -f docker-compose.test.yml up test-unit
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Performance Tips
|
||||
|
||||
### 1. Cache Management
|
||||
|
||||
```bash
|
||||
# Share Go cache between runs
|
||||
podman run -v $(pwd)/.gocache:/root/.cache/go-build golang:1.25-alpine go build .
|
||||
|
||||
# Clear cache when needed
|
||||
rm -rf .gocache .gomodcache
|
||||
```
|
||||
|
||||
### 2. Layer Caching
|
||||
|
||||
```bash
|
||||
# Dockerfiles use multi-stage builds for efficiency
|
||||
# Build cache is preserved between runs
|
||||
podman build -t mev-bot:latest . # Uses cache
|
||||
```
|
||||
|
||||
### 3. Memory & CPU
|
||||
|
||||
```bash
|
||||
# Run with resource limits
|
||||
podman run --cpus=4 --memory=4g golang:1.25-alpine go test ./...
|
||||
|
||||
# Increase memory for heavy operations
|
||||
podman compose -f docker-compose.test.yml run --memory=8g test-coverage
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## CI/CD Integration
|
||||
|
||||
### GitHub Actions (Example)
|
||||
|
||||
```yaml
|
||||
- name: Setup Podman
|
||||
run: |
|
||||
sudo apt-get update
|
||||
sudo apt-get install -y podman podman-compose
|
||||
|
||||
- name: Run Tests in Podman
|
||||
run: |
|
||||
podman compose -f docker-compose.test.yml up test-unit
|
||||
|
||||
- name: Generate Coverage
|
||||
run: |
|
||||
podman compose -f docker-compose.test.yml up test-coverage
|
||||
```
|
||||
|
||||
### Local CI Testing
|
||||
|
||||
```bash
|
||||
# Test your CI locally before pushing
|
||||
make ci-container quick
|
||||
|
||||
# Full CI pipeline
|
||||
make ci-full
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Migration from Docker
|
||||
|
||||
### If You Have Docker Installed
|
||||
|
||||
No action needed! Scripts automatically:
|
||||
1. Try Podman first (preferred)
|
||||
2. Fall back to Docker if needed
|
||||
3. Use correct compose command
|
||||
4. Work seamlessly with both
|
||||
|
||||
### Uninstalling Docker (Optional)
|
||||
|
||||
```bash
|
||||
# Docker is no longer required but can coexist
|
||||
# To remove:
|
||||
sudo apt-get remove -y docker.io docker-compose
|
||||
|
||||
# Keep Podman
|
||||
sudo apt-get install -y podman podman-compose
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Best Practices
|
||||
|
||||
### 1. Always Check Runtime
|
||||
|
||||
```bash
|
||||
# Before running scripts, verify:
|
||||
./scripts/container-runtime.sh status
|
||||
```
|
||||
|
||||
### 2. Use Makefiles
|
||||
|
||||
```bash
|
||||
# Makefile commands are portable
|
||||
make ci-container # Works with both
|
||||
```
|
||||
|
||||
### 3. Keep Scripts Updated
|
||||
|
||||
```bash
|
||||
# Scripts reference container-runtime.sh
|
||||
# Update this file if you customize the detection logic
|
||||
```
|
||||
|
||||
### 4. Monitor Resources
|
||||
|
||||
```bash
|
||||
# Check container resource usage
|
||||
podman stats
|
||||
|
||||
# Monitor processes
|
||||
podman top <container_id>
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Summary
|
||||
|
||||
| Aspect | Status | Notes |
|
||||
|--------|--------|-------|
|
||||
| **Podman Support** | ✅ Full | Primary runtime |
|
||||
| **Docker Support** | ✅ Full | Automatic fallback |
|
||||
| **DinD Support** | ✅ Full | Podman-in-Podman ready |
|
||||
| **Compose Files** | ✅ Compatible | Works with both runtimes |
|
||||
| **Scripts** | ✅ Updated | All use container-runtime.sh |
|
||||
| **Make Commands** | ✅ Working | Transparently use detected runtime |
|
||||
| **CI/CD Ready** | ✅ Yes | GitHub Actions compatible |
|
||||
|
||||
---
|
||||
|
||||
## Next Steps
|
||||
|
||||
1. ✅ Install Podman: `sudo apt-get install podman podman-compose`
|
||||
2. ✅ Verify: `./scripts/container-runtime.sh status`
|
||||
3. ✅ Run tests: `podman compose -f docker-compose.test.yml up test-unit`
|
||||
4. ✅ Use make: `make ci-container quick`
|
||||
|
||||
---
|
||||
|
||||
## Support
|
||||
|
||||
For issues:
|
||||
1. Check runtime: `./scripts/container-runtime.sh status`
|
||||
2. Test manually: `podman run --rm golang:1.25-alpine go version`
|
||||
3. Review this guide: Troubleshooting section above
|
||||
|
||||
Generated: 2025-11-06
|
||||
Configuration: Podman-first with Docker fallback
|
||||
Reference in New Issue
Block a user