feat: create v2-prep branch with comprehensive planning
Restructured project for V2 refactor: **Structure Changes:** - Moved all V1 code to orig/ folder (preserved with git mv) - Created docs/planning/ directory - Added orig/README_V1.md explaining V1 preservation **Planning Documents:** - 00_V2_MASTER_PLAN.md: Complete architecture overview - Executive summary of critical V1 issues - High-level component architecture diagrams - 5-phase implementation roadmap - Success metrics and risk mitigation - 07_TASK_BREAKDOWN.md: Atomic task breakdown - 99+ hours of detailed tasks - Every task < 2 hours (atomic) - Clear dependencies and success criteria - Organized by implementation phase **V2 Key Improvements:** - Per-exchange parsers (factory pattern) - Multi-layer strict validation - Multi-index pool cache - Background validation pipeline - Comprehensive observability **Critical Issues Addressed:** - Zero address tokens (strict validation + cache enrichment) - Parsing accuracy (protocol-specific parsers) - No audit trail (background validation channel) - Inefficient lookups (multi-index cache) - Stats disconnection (event-driven metrics) Next Steps: 1. Review planning documents 2. Begin Phase 1: Foundation (P1-001 through P1-010) 3. Implement parsers in Phase 2 4. Build cache system in Phase 3 5. Add validation pipeline in Phase 4 6. Migrate and test in Phase 5 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
Administrator
295
DEV_ENVIRONMENT.md
Normal file
295
DEV_ENVIRONMENT.md
Normal file
@@ -0,0 +1,295 @@
|
||||
# MEV Bot Development Environment
|
||||
|
||||
This guide explains how to use the development environment with branch selection and podman-in-podman support.
|
||||
|
||||
## Quick Start
|
||||
|
||||
### Start Development Environment
|
||||
|
||||
```bash
|
||||
# Start with default branch (master-dev)
|
||||
./scripts/dev-env.sh start
|
||||
|
||||
# Start with specific branch
|
||||
./scripts/dev-env.sh start fix-critical-arbitrage-bugs
|
||||
|
||||
# Start with master branch
|
||||
./scripts/dev-env.sh start master
|
||||
```
|
||||
|
||||
### Switch Branches
|
||||
|
||||
```bash
|
||||
# Switch to a different branch (stops current, checks out, rebuilds)
|
||||
./scripts/dev-env.sh switch feat-new-feature
|
||||
|
||||
# List available branches
|
||||
./scripts/dev-env.sh branches
|
||||
```
|
||||
|
||||
### View Logs
|
||||
|
||||
```bash
|
||||
# Follow logs
|
||||
./scripts/dev-env.sh logs -f
|
||||
|
||||
# Show last 100 lines
|
||||
./scripts/dev-env.sh logs --tail 100
|
||||
```
|
||||
|
||||
### Access Container Shell
|
||||
|
||||
```bash
|
||||
# Open interactive shell in running container
|
||||
./scripts/dev-env.sh shell
|
||||
```
|
||||
|
||||
### Stop/Restart
|
||||
|
||||
```bash
|
||||
# Stop development environment
|
||||
./scripts/dev-env.sh stop
|
||||
|
||||
# Restart with same branch
|
||||
./scripts/dev-env.sh restart
|
||||
|
||||
# Restart with different branch
|
||||
./scripts/dev-env.sh restart master
|
||||
```
|
||||
|
||||
### Rebuild
|
||||
|
||||
```bash
|
||||
# Full rebuild (no cache) with current branch
|
||||
./scripts/dev-env.sh rebuild
|
||||
|
||||
# Full rebuild with specific branch
|
||||
./scripts/dev-env.sh rebuild master-dev
|
||||
```
|
||||
|
||||
## Architecture
|
||||
|
||||
### Files
|
||||
|
||||
- **Dockerfile.dev** - Development dockerfile with branch selection support
|
||||
- **docker-compose.dev.yml** - Development compose configuration
|
||||
- **scripts/dev-env.sh** - Development environment management script
|
||||
|
||||
### Features
|
||||
|
||||
1. **Branch Selection**: Build and run from any git branch
|
||||
2. **Podman-in-Podman**: Container can run podman commands (privileged mode)
|
||||
3. **Live Logs**: Persistent log directory mounted from host
|
||||
4. **Hot Reload**: Source code mounted for development (optional)
|
||||
5. **Resource Management**: Configurable CPU and memory limits
|
||||
|
||||
## Environment Variables
|
||||
|
||||
Set these in `.env` or export before running:
|
||||
|
||||
```bash
|
||||
# Branch to build from
|
||||
export GIT_BRANCH=master-dev
|
||||
|
||||
# Logging level
|
||||
export LOG_LEVEL=debug
|
||||
|
||||
# Enable metrics
|
||||
export METRICS_ENABLED=true
|
||||
|
||||
# RPC endpoint
|
||||
export ARBITRUM_RPC_ENDPOINT=https://arb1.arbitrum.io/rpc
|
||||
```
|
||||
|
||||
## Podman-in-Podman Support
|
||||
|
||||
The development environment supports running podman inside the container:
|
||||
|
||||
```bash
|
||||
# Access container shell
|
||||
./scripts/dev-env.sh shell
|
||||
|
||||
# Inside container, you can use podman
|
||||
podman ps
|
||||
podman images
|
||||
podman run ...
|
||||
```
|
||||
|
||||
### Requirements
|
||||
|
||||
- Container runs in privileged mode
|
||||
- Podman socket mounted from host: `/run/podman/podman.sock`
|
||||
- Persistent storage volume: `podman-storage`
|
||||
|
||||
## Development Workflow
|
||||
|
||||
### Feature Development
|
||||
|
||||
```bash
|
||||
# 1. Create feature branch
|
||||
git checkout -b feat-my-feature
|
||||
|
||||
# 2. Start development environment with your branch
|
||||
./scripts/dev-env.sh start feat-my-feature
|
||||
|
||||
# 3. Make changes in your local files
|
||||
|
||||
# 4. Rebuild to test changes
|
||||
./scripts/dev-env.sh rebuild feat-my-feature
|
||||
|
||||
# 5. View logs to verify
|
||||
./scripts/dev-env.sh logs -f
|
||||
|
||||
# 6. When done, stop environment
|
||||
./scripts/dev-env.sh stop
|
||||
```
|
||||
|
||||
### Testing Different Branches
|
||||
|
||||
```bash
|
||||
# Test fix branch
|
||||
./scripts/dev-env.sh switch fix-critical-arbitrage-bugs
|
||||
|
||||
# Test feature branch
|
||||
./scripts/dev-env.sh switch feat-podman-compose-support
|
||||
|
||||
# Go back to master-dev
|
||||
./scripts/dev-env.sh switch master-dev
|
||||
```
|
||||
|
||||
### Debugging
|
||||
|
||||
```bash
|
||||
# Start with debug logging
|
||||
export LOG_LEVEL=debug
|
||||
./scripts/dev-env.sh start
|
||||
|
||||
# Access container for inspection
|
||||
./scripts/dev-env.sh shell
|
||||
|
||||
# Inside container:
|
||||
ps aux # Check processes
|
||||
cat /app/config/config.yaml # View config
|
||||
ls -la /app # List files
|
||||
```
|
||||
|
||||
## Comparison: Development vs Production
|
||||
|
||||
| Feature | Development (dev-env.sh) | Production (deploy-production-docker.sh) |
|
||||
|---------|-------------------------|------------------------------------------|
|
||||
| Branch Selection | ✅ Yes (any branch) | ✅ Yes (defaults to master) |
|
||||
| Dockerfile | Dockerfile.dev | Dockerfile |
|
||||
| Compose File | docker-compose.dev.yml | docker-compose.yml |
|
||||
| Privileged Mode | ✅ Yes (for podman-in-podman) | ❌ No |
|
||||
| Resource Limits | Higher (4 CPU, 4GB RAM) | Lower (2 CPU, 2GB RAM) |
|
||||
| Restart Policy | unless-stopped | always |
|
||||
| Container Name | mev-bot-dev-{branch} | mev-bot-production |
|
||||
| Source Mount | ✅ Yes (optional) | ❌ No |
|
||||
| Auto-start on Boot | ❌ No | ✅ Yes (via systemd) |
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Container Won't Start
|
||||
|
||||
```bash
|
||||
# Check status
|
||||
./scripts/dev-env.sh status
|
||||
|
||||
# View logs
|
||||
./scripts/dev-env.sh logs
|
||||
|
||||
# Rebuild from scratch
|
||||
./scripts/dev-env.sh clean
|
||||
./scripts/dev-env.sh rebuild
|
||||
```
|
||||
|
||||
### Branch Doesn't Exist
|
||||
|
||||
```bash
|
||||
# List available branches
|
||||
./scripts/dev-env.sh branches
|
||||
|
||||
# Update branch list
|
||||
git fetch --all
|
||||
./scripts/dev-env.sh branches
|
||||
```
|
||||
|
||||
### Podman Socket Not Available
|
||||
|
||||
```bash
|
||||
# Check if podman socket is running
|
||||
systemctl --user status podman.socket
|
||||
|
||||
# Start podman socket
|
||||
systemctl --user start podman.socket
|
||||
```
|
||||
|
||||
### Out of Disk Space
|
||||
|
||||
```bash
|
||||
# Clean old images and containers
|
||||
./scripts/dev-env.sh clean
|
||||
|
||||
# Prune podman system
|
||||
podman system prune -a
|
||||
```
|
||||
|
||||
## Advanced Usage
|
||||
|
||||
### Custom Configuration
|
||||
|
||||
Create a `.env.dev` file for development-specific settings:
|
||||
|
||||
```bash
|
||||
# .env.dev
|
||||
GIT_BRANCH=master-dev
|
||||
LOG_LEVEL=debug
|
||||
METRICS_ENABLED=true
|
||||
METRICS_PORT=9090
|
||||
PORT=8080
|
||||
```
|
||||
|
||||
Load it before starting:
|
||||
|
||||
```bash
|
||||
set -a && source .env.dev && set +a
|
||||
./scripts/dev-env.sh start
|
||||
```
|
||||
|
||||
### Manual Podman Commands
|
||||
|
||||
```bash
|
||||
# Use docker-compose directly
|
||||
export GIT_BRANCH=feat-my-branch
|
||||
podman-compose -f docker-compose.dev.yml build
|
||||
podman-compose -f docker-compose.dev.yml up -d
|
||||
|
||||
# Or use single command
|
||||
GIT_BRANCH=feat-my-branch podman-compose -f docker-compose.dev.yml up -d --build
|
||||
```
|
||||
|
||||
### Performance Monitoring
|
||||
|
||||
```bash
|
||||
# Access metrics endpoint (if enabled)
|
||||
curl http://localhost:9090/metrics
|
||||
|
||||
# View resource usage
|
||||
podman stats
|
||||
```
|
||||
|
||||
## Best Practices
|
||||
|
||||
1. **Use feature branches** for development, not master
|
||||
2. **Test in dev environment** before merging to master-dev
|
||||
3. **Rebuild after git pull** to ensure latest code
|
||||
4. **Monitor logs** during development with `-f` flag
|
||||
5. **Clean regularly** to free up disk space
|
||||
6. **Use specific branches** instead of relying on defaults
|
||||
|
||||
## Related Documentation
|
||||
|
||||
- [DEPLOYMENT.md](DEPLOYMENT.md) - Production deployment guide
|
||||
- [CLAUDE.md](CLAUDE.md) - Project overview and guidelines
|
||||
- [docker-compose.dev.yml](docker-compose.dev.yml) - Development compose configuration
|
||||
- [Dockerfile.dev](Dockerfile.dev) - Development dockerfile
|
||||
Reference in New Issue
Block a user