Backend patterns for AI work

A request handler that calls a model inline blocks for seconds, can't be retried safely, and loses its work if the process restarts. The fix is to treat AI work as a durable job and move it off the request path.

AI work is a job, not a function call

The request creates a row, returns 202 Accepted, and a worker does the work. The job's state lives in Postgres so a client can poll for it and an operator can inspect it.

POST /jobs/{id}/process   ->  202, job stays queued

queued -> dispatching -> running -> succeeded
                                 \-> failed

running means a worker took ownership — the route never sets it.
succeeded / failed are final and worker-owned.

Persist attempt metadata with the job — attempt count, started_at, completed_at, token usage, cost — so a stuck or repeatedly-failing job is debuggable from the row alone.

The dual-write problem

The route needs to do two things: record the job in Postgres and enqueue it in Redis. If the DB write succeeds and the enqueue fails, the job exists but nothing will ever run it.

• First mitigation: an explicit dispatching state. If the enqueue throws, roll the job back to queued and return 503. The gap is visible and recoverable.
• Stronger fix — the outbox pattern: the route writes the job state and an outbox_events row in one Postgres transaction. A separate dispatcher process reads pending events and delivers them to the queue, retrying independently. Business state and delivery intent commit together or not at all.

API        writes job + outbox event   (one transaction)
dispatcher reads pending events -> enqueues -> marks delivered / failed
worker     pulls from queue -> runs job -> writes result to Postgres

Postgres is the source of truth. Redis is delivery.

Idempotency keys

A client that times out will retry a POST it isn't sure landed. Without protection, that creates a second job. With an Idempotency-Key header, the server hashes method + path + body against the stored key: same key and same request replays the original response; same key and a different request is a 409.

Workers own their own resources

A background worker opens its own database session and builds its own repositories and service objects inside the worker boundary. Request-scoped dependencies must not leak into background work — their session lifecycle is tied to a request that has already returned.

Provider calls need budgets

• Explicit timeout per call — set at the client/adapter boundary, not scattered through services.
• Bounded retries — use the provider client's retry system, cap it (e.g. ≤ 5), and validate the config at startup. Unbounded retries multiply cost and hide outages.
• A stable adapter interface — services depend on SummaryProcessor / EmbeddingProvider, never on a vendor SDK class. Tests use fakes and never hit the network by default.

Health vs readiness

GET /health     process is alive          — do NOT fail it on a dependency outage
GET /readiness  can serve real traffic     — checks Postgres (SELECT 1), Redis (PING)
                                              returns 503 if a required dep is down

Logs you can query

Structured JSON logs with a per-request X-Request-ID carried through contextvars, so one failing call can be traced end to end: method, path, status, duration, request id, and — for AI jobs — job id, status, attempt count, timestamps.

The recurring rule: the API accepts and records work; the worker executes it; Postgres holds the truth; Redis only moves messages. Every failure mode above is a version of "what happens if this step succeeds but the next one doesn't".