Long-Running AI Work Is a Durable Workflow - Production AI Systems - Part 4
Long-running AI work should be modeled as durable state transitions with idempotent stages, checkpoints, bounded retries, cancellation, and backpressure.
Part 4 of 5 in Production AI Systems. Part 3 established how generated output crosses a layered trust boundary.
A support-case-summary request begins with document parsing, continues through retrieval and generation, may require repair, and can stop for human review. Keeping the original HTTP request open does not make those stages one atomic operation. It only ties durable business work to a fragile connection.
Clients disconnect. Proxies impose timeouts. Workers restart. Providers rate-limit. A reviewer may respond tomorrow. Retrying the request can duplicate expensive work or repeat a side effect.
Long-running AI work is a durable state machine, not a slow function call.
The design changes when the job record—not the worker or queue message—becomes the source of truth.
Accept work, then expose its state
For work that cannot reliably complete within an interactive latency budget, the API should accept a job and return its identity.
sequenceDiagram
actor U as Client
participant A as API
participant D as Job Store
participant Q as Queue
participant W as Worker
U->>A: Start support-case summary
A->>D: Create queued job
A->>Q: Publish job reference
A-->>U: 202 Accepted and job ID
Q->>W: Deliver job reference
W->>D: Claim queued job
W->>W: Execute next durable stage
W->>D: Persist checkpoint and state
RFC 9110 defines 202 Accepted for a request accepted for processing whose work has not completed. The response should point to a status resource rather than pretend that acceptance means success.
The job store records tenant, workflow version, state, attempt, current stage, progress, heartbeat, result reference, error classification, and provenance. The queue transports a job reference to workers. If the queue loses or duplicates a delivery, the job record still answers what should happen next.
Model the state transitions explicitly
An explicit state machine prevents status fields from becoming arbitrary labels.
stateDiagram-v2
[*] --> Queued
Queued --> Running: Claimed
Queued --> Cancelled: Cancel before start
Running --> Retrying: Retryable failure
Retrying --> Queued: Delay elapsed
Running --> WaitingForInput: Review required
WaitingForInput --> Queued: Input received
Running --> Completed: Result committed
Running --> Failed: Terminal failure
Running --> Cancelling: Cancellation requested
Cancelling --> Cancelled: Safe checkpoint reached
Completed --> [*]
Failed --> [*]
Cancelled --> [*]
Only declared transitions are allowed. A worker should claim queued → running with a conditional update so two deliveries cannot both become the active owner. Completion should persist the result and terminal state together where the storage model permits it.
The workflow version matters because a job can outlive a deployment. A resumed job must either continue under compatible semantics or be migrated explicitly. Letting a new worker reinterpret old state silently is a hidden production change.
At-least-once delivery moves correctness into the consumer
Many practical queue systems can deliver a message more than once. AWS documents this behavior for standard SQS queues and recommends idempotent consumers. The general consequence is more important than the product: a worker must assume that any delivery may be a replay.
Imagine a worker persists a completed support-case summary and crashes before acknowledging its queue message. The message returns. A correct worker reads the job, sees the terminal state, and acknowledges without generating again. A worker that treats delivery as proof of pending work repeats cost and may produce a different result.
Idempotency has several layers:
- A unique operation key prevents equivalent start requests from creating duplicate jobs.
- Conditional state transitions prevent multiple workers from owning one stage.
- Stage-execution records prevent completed stages from replaying.
- External side effects use provider-supported idempotency keys or a durable application ledger.
- Result writes use stable identifiers or uniqueness constraints.
The key must represent business intent, not merely one network request. Two requests for the same support case, evidence-bundle version, workflow version, and schema may be equivalent; the same support case with a new evidence bundle is not.
Stages create recovery boundaries
The support-case workflow can be decomposed into parse, normalize, retrieve, generate, validate, review, store, and notify stages. Each stage declares its input, output, timeout, retry policy, idempotency boundary, version, and checkpoint.
flowchart LR
P[Parse] --> A[(Parsed Artifact)]
A --> R[Retrieve Evidence]
R --> B[(Evidence Set)]
B --> G[Generate]
G --> C[(Raw Output)]
C --> V[Validate]
V --> D[(Disposition)]
Persisting expensive intermediate artifacts means a validation failure does not repeat parsing and generation unnecessarily. Checkpoints also make failures inspectable. A raw model output and validation report can be reviewed without reconstructing the exact provider response from partial logs.
Checkpoint reuse requires version compatibility. Parsed artifacts should identify parser and source-document versions. Retrieved evidence should identify the index and query configuration. Generated output should identify prompt, model, and schema. A stale checkpoint is faster only until it causes an incorrect result.
Retry the failing boundary, not the whole workflow
Retries should be stage-specific and budgeted.
| Failure | Retry? | Default action |
|---|---|---|
| Provider rate limit | Yes | Back off with jitter within workflow budget |
| Temporary network failure | Yes | Retry the current idempotent stage |
| Corrupt source document | No | Mark invalid input |
| Authorization failure | No | Fail securely and audit |
| Invalid generated structure | Maybe | Bounded regenerate or repair |
| Unsupported evidence claim | No blind retry | Review, reject, or regenerate from source |
| Workflow budget exceeded | No | Fail with explicit budget outcome |
A retry budget should bound attempts, elapsed time, and spend. A queue's redelivery count is not enough because one attempt may contain several provider calls. The job needs a workflow-level budget that each stage consumes.
After exhaustion, move the job to a visible terminal or review state. A dead-letter queue can retain undeliverable messages, but it should not become the business source of truth. Operations need the job ID, stage, error class, attempts, workflow version, trace ID, and an explicit replay procedure.
Waiting, cancellation, and backpressure are ordinary states
Human review is not a worker sleeping for hours. The workflow persists a checkpoint, enters waiting_for_input, and releases compute. A review task carries a deadline, assignee or queue, required evidence, and the workflow version it applies to. When input arrives, the workflow validates that it is still current before resuming.
Cancellation is cooperative. A worker checks cancellation between stages and before expensive calls or side effects. Work already sent to an external provider may not stop immediately, so cancelling and cancelled are distinct from pretending every operation is interruptible.
Progress should report meaningful units: documents parsed, records validated, or the current stage. An invented percentage for an unpredictable model call creates false precision.
Backpressure protects the dependencies
Adding workers can overwhelm model quotas, parsers, databases, or accelerators. Admission control and concurrency limits belong before saturation, not after queue depth becomes an incident.
Interactive, batch, and evaluation work often need separate capacity. A large offline evaluation should not consume every provider token or GPU slot needed for user-facing summaries. Priority queues help only when downstream concurrency and quota reservations enforce the priority.
Autoscaling from queue depth alone can also amplify overload. Useful signals include oldest-job age, stage duration, dependency saturation, failure rate, and the capacity actually available at constrained downstream services.
Those signals keep throughput decisions tied to the real bottleneck.
Choose the smallest orchestration style that fits
Queue chaining is adequate for short, stable pipelines where each stage independently publishes the next. A central durable orchestrator is clearer when one owner must enforce the state machine. A workflow engine becomes valuable for multi-day execution, timers, compensation, human waits, and many branching paths.
The choice should follow workflow semantics rather than fashion. A three-stage extraction does not automatically need a new platform. Conversely, reimplementing timers, replay, and durable waits inside ad hoc queue consumers is rarely the simpler system.
Whatever the mechanism, workers must remain replaceable. Durable state belongs in the job store and checkpoints, never only in process memory.
Design packet: job and stage review
The canonical job record answers:
Who owns this work?
Which workflow and input versions apply?
What state and stage is authoritative?
Who currently holds the lease or claim?
Which checkpoints exist?
What retry, time, and cost budgets remain?
What result or failure was committed?
How can the run be traced and reviewed?
Every stage then uses the same failure review:
- Did the stage start from a compatible checkpoint?
- Can another delivery repeat it safely?
- Is the failure transient, terminal, or uncertain?
- What bounded retry is allowed?
- Which state is persisted after exhaustion?
- Could cancellation or a stale worker still produce a side effect?
The workflow can now survive disconnects and restarts, but durable execution does not establish quality. The final article closes the architecture with evaluation: the control system that decides whether a change is safe to release.