mev-beta/docs/6_operations/SECRETS_MANAGEMENT.md

Secrets Management Strategy

This guide defines how secrets move through every environment for the MEV Bot. It is now the source of truth for operators, CI, and fellow agents when handling API keys, private keys, TLS material, and configuration values.

Summary

Environment	Storage Backend	Access Method	Rotation Cadence	Notes
Local development	.env.local / .env.devenv (gitignored) generated from env/templates/*.env	make init-env or manual copy	On demand	Never commit generated files. Default values are placeholders only.
CI (Drone, Harness)	AWS SSM Parameter Store (/mev-beta/<env>/<name>)	Pipeline IAM role + aws ssm get-parameter wrapper in CI scripts	30 days	Parameters are encrypted with KMS key alias/mev-beta-secrets.
Staging	AWS SSM Parameter Store (same hierarchy)	Harness service account role, read‑only	30 days	Harness pipelines sync secrets into staging pods via scripts/render-config.sh.
Production	HashiCorp Vault (namespace mev/prod)	Vault Agent sidecar + AppRole login (role_id vaulted in SSM)	14 days for API keys, 7 days for signing keys	Vault is the source of truth. SSM only stores bootstrap AppRole credentials.

Vault Layout (Production)

• Mount: secret/mev/prod/
• Paths:
  • secret/mev/prod/rpc/{provider} – HTTPS RPC credentials and rotation metadata.
  • secret/mev/prod/keystore/main – Encrypted MEV bot keystore bundle (GCM sealed).
  • secret/mev/prod/webhooks/{service} – Alerting/webhook tokens.
  • secret/mev/prod/infra/harness – Harness deploy tokens (read-only by Deploy role).
• Policies:
  • mev-bot-runtime – read secret/mev/prod/* (excluding /infra).
  • mev-deployer – read all + update deploy tokens.
  • mev-operations – read/write all but requires multi-factor.

Runtime Retrieval Flow

1. Harness deploy job injects Vault Agent sidecar with AppRole credentials (stored in SSM).
2. Sidecar renders secrets into /run/secrets/* using template file config/vault/runtime.hcl.
3. Bot reads MEV_BOT_KEYSTORE_PATH and other environment variables from rendered files.
4. Agent renews leases automatically; restarts occur if lease renewal fails.

AWS SSM Hierarchy (CI & Staging)

/mev-beta/
  staging/
    arbitrum/rpc_url
    arbitrum/ws_url
    mev/security_webhook
  ci/
    drone/github_token
    drone/harness_token

• Values are encrypted with customer-managed KMS key (alias/mev-beta-secrets).
• Drone CLI wrapper scripts/ci/export_ssm.sh exports required values at runtime.
• Harness pipeline uses the same helper via scripts/ci/harness_env.sh.

Local Development Flow

1. Copy template: cp env/templates/.env.devenv env/.env.local.
2. Fill required fields manually (Infura API key, test private key, etc.).
3. cmd/mev-bot loads .env.local automatically when GO_ENV=development.
4. Never commit generated files—.gitignore already covers everything under env/.

Rotation Playbooks

Vault API Key Rotation

1. Operator authenticates with Vault MFA (vault login -method=oidc).
2. Run scripts/secrets/rotate_vault_key.sh --path secret/mev/prod/rpc/primary.
3. Script writes new value, updates rotation metadata, and notifies Slack via webhook.
4. Confirm bot picked up new credentials via kubectl logs (look for Vault secret refresh).

SSM Parameter Rotation

1. Use scripts/secrets/rotate_ssm.sh staging arbitrum/rpc_url.
2. Harness staging pipeline automatically pulls new value on next deployment.

Keystore Rotation

1. Generate new key with scripts/keys/create_keystore.sh.
2. Upload encrypted archive to Vault path secret/mev/prod/keystore/main.
3. Update MEV_BOT_KEYSTORE_PATH pointer in Vault template.
4. Trigger rolling restart (kubectl rollout restart deployment/mev-bot).

Auditing & Compliance

• All secret reads are logged:
  • Vault audit backend (file + syslog).
  • AWS CloudTrail for SSM + KMS.
• Weekly report generated via scripts/secrets/audit_report.sh, stored in reports/security/<date>_secrets_audit.json.
• Any ad-hoc access must be recorded in docs/8_reports/security_incident_log.md.

Operational Guardrails

• Never embed secrets in Kubernetes manifests, Compose files, or config YAML checked into git.
• Use config/providers.yaml placeholders only; runtime values always come from Vault/SSM.
• Secrets destined for CI must go through the helper scripts—manual aws ssm put-parameter calls are forbidden to keep rotation metadata consistent.
• Keep MEV_BOT_ENCRYPTION_KEY out of local shells when screen-sharing. Use aws ssm get-parameter --with-decryption piped directly into files.

Next Steps

• Automate Vault Agent deployment manifest for production (tracked separately).
• Add staging smoke test that verifies SSM sync prior to deployment.