Background Jobs in Python — arq, Dramatiq, Taskiq, and Celery in 2026

Every non-trivial Python service eventually needs background jobs. Email, webhooks, ML inference, scheduled cleanup, data exports. In 2026 the choices are clearer than ever — but the right one depends on what you’re building.

This post is the working comparison. Four systems, when each wins, and the production patterns that apply to all of them.

The four systems

Pick by codebase shape:

• Already async (FastAPI, asyncio)? → arq.
• Sync app, want simplicity? → Dramatiq.
• Want typed tasks across multiple brokers? → Taskiq.
• Have a Celery codebase that works? → Stay.

arq — async-native, Redis-backed

The friendliest choice for async Python in 2026. Tiny dependency surface, just-Redis, and tasks are async functions:

# tasks.py
from arq import create_pool
from arq.connections import RedisSettings


async def send_welcome_email(ctx, user_id: int):
    # ctx contains the Redis pool, retry count, etc.
    user = await fetch_user(user_id)
    await send_email(user.email, "Welcome!", template="welcome")


class WorkerSettings:
    functions = [send_welcome_email]
    redis_settings = RedisSettings(host="localhost", port=6379)
    max_jobs = 32                    # concurrent jobs per worker
    job_timeout = 60
    max_tries = 5

Run the worker:

arq tasks.WorkerSettings

Enqueue from FastAPI:

# app/main.py
from arq import create_pool
from contextlib import asynccontextmanager


@asynccontextmanager
async def lifespan(app):
    app.state.arq = await create_pool(RedisSettings())
    yield
    await app.state.arq.close()


app = FastAPI(lifespan=lifespan)


@app.post("/users")
async def create_user(payload: UserCreate, request: Request):
    user = await db.create_user(payload)
    await request.app.state.arq.enqueue_job("send_welcome_email", user.id)
    return user

What’s good:

• Async all the way down. No sync_to_async shenanigans.
• Cron jobs built in:

from arq.cron import cron

class WorkerSettings:
    cron_jobs = [cron(daily_cleanup, hour=3, minute=0)]

• Defer + delay + retry with sensible defaults.
• Tiny. ~3000 LOC. You can read it.

What’s missing vs Celery:

• Only Redis. No RabbitMQ / SQS native support.
• Smaller ecosystem (no zillion plugins).

For most async-first FastAPI apps, arq is the right default. See the patterns in FastAPI + Pydantic v2 + SQLAlchemy 2.0 .

Dramatiq — boring and reliable

Dramatiq is sync-first, RabbitMQ-or-Redis, opinionated about reliability. It’s the “Erlang-style” Python job library: simple semantics, hard to misuse.

# tasks.py
import dramatiq
from dramatiq.brokers.rabbitmq import RabbitmqBroker

dramatiq.set_broker(RabbitmqBroker(host="rabbitmq"))


@dramatiq.actor(max_retries=5, time_limit=60_000)
def send_welcome_email(user_id: int):
    user = fetch_user_sync(user_id)
    send_email(user.email, "Welcome!", template="welcome")

dramatiq tasks

Enqueue:

send_welcome_email.send(42)

What’s good:

• Reliable defaults. Acks-on-success, message redelivery, exponential backoff retries.
• Middleware system for adding logging, metrics, rate limiting.
• RabbitMQ-first, which is the broker most enterprises already have.
• Mature. Production-tested over years.

What’s missing:

• Async support is via wrappers (dramatiq-tasks-asyncio); not first-class.
• Smaller community than Celery.

For sync Python apps that want a no-drama job system, Dramatiq is the right call. For async apps, you’ll feel the seams.

Taskiq — typed across brokers

Taskiq is the newest serious entrant. Async-first, typed via type hints, and supports many brokers:

# tasks.py
from taskiq import TaskiqScheduler
from taskiq_redis import ListQueueBroker, RedisAsyncResultBackend

broker = ListQueueBroker("redis://localhost:6379").with_result_backend(
    RedisAsyncResultBackend("redis://localhost:6379")
)


@broker.task(retry_on_error=True, max_retries=5)
async def send_welcome_email(user_id: int) -> str:
    user = await fetch_user(user_id)
    await send_email(user.email, "Welcome!", template="welcome")
    return f"sent to {user.email}"

taskiq worker tasks:broker

Enqueue:

task = await send_welcome_email.kiq(42)
result = await task.wait_result(timeout=10)
print(result.return_value)               # "sent to [email protected]" — typed

What’s good:

• Typed return values. The result type is the function’s return type.
• Multi-broker. Redis, RabbitMQ, NATS, Kafka, AMQP, in-memory for tests.
• Async + sync native.
• Middlewares for distributed tracing, dead-letter, etc.

What’s missing:

• Smaller ecosystem than Celery / arq.
• Some maturity edges still being polished.

For greenfield projects that want typed tasks and flexibility on the broker, Taskiq is the clean pick.

Celery — when to keep it

Celery is the Python job system. 2010-era design, still everywhere, still works.

from celery import Celery

app = Celery("tasks", broker="redis://localhost:6379")


@app.task(bind=True, max_retries=5, default_retry_delay=30)
def send_welcome_email(self, user_id):
    try:
        user = fetch_user_sync(user_id)
        send_email(user.email, "Welcome!", template="welcome")
    except TransientError as exc:
        raise self.retry(exc=exc)

When Celery still wins:

• Existing codebase. Don’t migrate “for fashion.”
• Plugin ecosystem. Celery Flower for monitoring, Celery Beat for scheduling, dozens of integrations.
• Multi-broker support is mature (you can switch RabbitMQ → SQS → Redis without much change).
• Battle-tested. Every weird production failure has been debugged elsewhere.

What’s painful:

• Async story is uneven. 5.x added async tasks but config is fiddly.
• Configuration surface is huge.
• Performance is fine, not great.

For new async apps, I would not pick Celery. For existing Celery deployments that work, leave them alone.

Production patterns (apply to all)

1. Idempotency

Every task should be safe to run twice. The minimum pattern:

async def process_payment(ctx, payment_id: str):
    # Check if we already handled this payment
    if await db.fetchval("SELECT 1 FROM processed_payments WHERE id = $1", payment_id):
        return
    # ... process ...
    await db.execute("INSERT INTO processed_payments (id) VALUES ($1)", payment_id)

See Idempotency, Retries, and Exactly-Once Illusions for the deeper patterns.

2. Dead-letter queues

Tasks that fail after max_retries should go somewhere — not vanish.

# arq
class WorkerSettings:
    on_job_end = lambda ctx, *args: log_to_dead_letter(ctx) if ctx["job_try"] >= 5 else None

# Dramatiq
@dramatiq.actor(max_retries=5)
def t(...):
    ...
# Failed messages auto-routed to a `xxx.dlq` queue — read and inspect.

A weekly review of the DLQ catches silent business failures that retries can’t fix.

3. Idempotency keys for producers

Same logical event publishes once. Use the outbox pattern when crossing service boundaries.

4. Observability

Every task should have:

• A unique ID (the broker’s plus your own correlation ID).
• A start log + end log + duration.
• Tracing spans (OpenTelemetry — see OpenTelemetry End-to-End ).
• Failure logs with full traceback.

Without this, “why did this user not get their email?” is unanswerable.

5. Bound concurrency per worker

If a downstream API rate-limits you, your job system shouldn’t barge in with 100 concurrent calls:

# arq
class WorkerSettings:
    max_jobs = 8                # concurrency limit per worker

Per-task limits via semaphore patterns inside the task body. See Modern AsyncIO Patterns .

6. Scheduling — separately deploy and monitor

Cron jobs (arq cron, Periodiq, Celery Beat, TaskiqScheduler) live in a separate process. Run only one scheduler per cluster or you’ll fire jobs N times. Use a leader election or single-replica deployment.

7. Graceful shutdown

When the worker pod terminates:

• Stop accepting new tasks.
• Finish in-flight tasks (within a deadline).
• Don’t ack messages until they’re processed.

All four systems handle this; verify your deploy config (e.g., terminationGracePeriodSeconds: 60 in Kubernetes).

Choosing — a quick rule

For a brand-new async Python service:

• Need a job queue, want minimum tooling? → arq. Redis-only, async, done.
• Need many brokers (RabbitMQ + Kafka + …) and typed tasks? → Taskiq.
• Sync app, want zero drama? → Dramatiq with RabbitMQ.
• Already have Celery and it works? → Stay.

Anti-patterns

1. Using FastAPI’s BackgroundTasks for real work

@app.post("/signup")
async def signup(payload: SignUp, bg: BackgroundTasks):
    user = await create_user(payload)
    bg.add_task(send_welcome_email, user.id)
    return user

This works for one-shot, fire-and-forget work that takes <100ms. It does not retry, does not persist across restarts, does not scale beyond one process. For anything more, use a real job system.

2. Long synchronous DB access in async tasks

async def process(ctx, id):
    user = User.objects.get(id=id)              # ⛔ sync ORM blocks event loop

Either use async ORM (asyncpg, async SQLAlchemy) or run sync in a threadpool (run_in_executor).

3. Heavy CPU work in the same worker pool

CPU-bound tasks (image resize, ML inference) starve the I/O loop. Run them in separate workers, separate queues, separate processes.

4. No retry strategy

A task that fails permanently with no backoff is a thundering herd against the underlying problem. Always: exponential backoff + jitter + max retries + DLQ.

5. Storing big payloads in the broker

The broker is for the envelope. Big payloads go in S3 or Postgres; the message references the ID. A 10 MB JSON message will misbehave in any broker.

Read this next

• Modern AsyncIO Patterns — TaskGroup, anyio — concurrency primitives jobs lean on.
• Idempotency, Retries, and Exactly-Once Illusions — the correctness patterns.
• Kafka vs NATS vs RabbitMQ — broker decisions.
• Celery Background Tasks Explained — the Celery-specific deep dive.

If you want a working FastAPI + arq + Postgres outbox starter with retries, DLQ, and OTel tracing, it’s at rajpoot.dev .

Building something AI-, backend-, or data-heavy and want a second pair of eyes? I do consulting and freelance work — see my projects and ways to reach me at rajpoot.dev .