Chapter 7: Alembic in CI. Drift detection, round-trip tests, deploy gates, pre-merge checks.

CI fundamentals

Run Alembic against a fresh DB in CI. Catches:

  • Migration syntax errors.
  • Column / type mismatches.
  • Missing imports in env.py.
  • Drift between models and migrations.

Postgres in CI

# GitHub Actions
services:
  postgres:
    image: postgres:17
    env:
      POSTGRES_PASSWORD: test
      POSTGRES_DB: test
    ports: ["5432:5432"]
    options: >-
      --health-cmd "pg_isready -U postgres"
      --health-interval 10s

env:
  DATABASE_URL: postgresql+asyncpg://postgres:test@localhost:5432/test

Or testcontainers. PG service is faster for CI.

Drift detection

Test that models and migrations are in sync:

# tests/test_alembic.py
from alembic.autogenerate import compare_metadata
from alembic.migration import MigrationContext

def test_no_drift(engine):
    # First, apply all migrations
    cfg = Config("alembic.ini")
    cfg.set_main_option("sqlalchemy.url", str(engine.url))
    command.upgrade(cfg, "head")
    
    # Then compare
    with engine.connect() as conn:
        ctx = MigrationContext.configure(conn)
        diff = compare_metadata(ctx, Base.metadata)
        assert diff == [], f"Drift: {diff}"

If diff is non-empty: someone changed models without generating a migration. Test fails.

Round-trip test

Verify migrations work both ways:

def test_migrate_up_down(engine):
    cfg = Config("alembic.ini")
    cfg.set_main_option("sqlalchemy.url", str(engine.url))
    
    command.upgrade(cfg, "head")
    command.downgrade(cfg, "base")
    command.upgrade(cfg, "head")

If any downgrade is broken or non-reversible: caught here.

For prod (forward-only) it’s still useful — downgrades are dev/test convenience.

Single-head check

- run: |
    if [ $(alembic heads | wc -l) -gt 1 ]; then
      echo "Multiple heads"
      exit 1
    fi

Block PRs that introduce a branch without a merge.

Dry-run

alembic upgrade head --sql > migration.sql

Prints SQL without applying. Useful for review.

Migration linting

For consistent migration style:

# tests/test_migration_style.py
import os

def test_all_migrations_have_message():
    for fname in os.listdir("migrations/versions"):
        if not fname.endswith(".py"): continue
        with open(f"migrations/versions/{fname}") as f:
            content = f.read()
        assert '"""' in content  # has docstring
        # ... other checks

Or use alembic-utils for view / function management with linting.

Pre-merge checks

GitHub Actions workflow:

name: Migrations
on: [pull_request]

jobs:
  test:
    runs-on: ubuntu-latest
    services:
      postgres:
        image: postgres:17
        # ...
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: { python-version: "3.13" }
      - run: uv sync --frozen
      - run: uv run alembic upgrade head
      - run: uv run pytest tests/test_alembic.py
      - run: |
          if [ $(uv run alembic heads | wc -l) -gt 1 ]; then exit 1; fi

Block merge until all migrations green.

Apply on deploy

K8s Job runs migrations before app rolls out:

apiVersion: batch/v1
kind: Job
metadata:
  name: migrate-{{ .Values.image.tag }}
  annotations:
    helm.sh/hook: pre-install,pre-upgrade
    helm.sh/hook-weight: "0"
spec:
  template:
    spec:
      containers:
        - name: migrate
          image: myapp:{{ .Values.image.tag }}
          command: ["alembic", "upgrade", "head"]
          envFrom:
            - secretRef: { name: myapp-secrets }
      restartPolicy: OnFailure
  backoffLimit: 3

Helm hook ensures Job runs before pod rollout. Job idempotent — Alembic skips already-applied.

Migration timing

For schemas that take long: run as a separate Job; deploy app after Job completes.

For really long: schedule for off-hours; communicate to ops.

Rollback strategy

Production: don’t downgrade. Instead, roll forward with a corrective migration.

If you must roll back:

  1. Stop traffic.
  2. alembic downgrade -1.
  3. Roll back app deploy.
  4. Investigate.

For most cases: roll forward is safer.

Migration history snapshot

For audit: every applied migration logged:

SELECT * FROM alembic_version;
-- one row: current head

For history of who-applied-when: extend the version table or use deploy logs / audit log.

Common mistakes

1. CI without DB

Migration tests must run against a real DB.

2. No drift test

Models change; migrations forgotten; deploys break.

3. Forgetting to commit lock changes

If you generate a migration but don’t commit migrations/versions/...py — CI catches; PR may not.

4. Migrating in app startup with multiple replicas

Race conditions; conflicting locks. Use a Job.

5. Long migrations blocking deploy

CI passes; production migration takes 2 hours; deploy stuck. Detect long migrations in staging first.

What’s next

Chapter 8: production patterns and recovery.

Read this next

Building something AI-, backend-, or data-heavy and want a second pair of eyes? I do consulting and freelance work — see my projects and ways to reach me at rajpoot.dev .