Make Agent Tool Retries Safe With Typed Contracts and Idempotency
Understand idempotency as a semantic contract for ambiguous tool outcomes, then prove durable replay and conflict handling locally.
An agent asks a tool to create a support ticket. The database commits, but the connection fails before the result reaches the agent. The run now faces an ambiguous outcome. Retrying may create a duplicate; stopping may abandon work that already succeeded.
The common response, “add retries”, solves transport reliability while making business correctness worse. The deeper problem is not whether the client retries. It is how the mutation defines identity, intent, commit, and recovery.
This article develops that contract, then uses a small SQLite experiment to prove four invariants: an equivalent retry returns the original result, changed intent fails closed, a response lost after commit creates no duplicate, and recovery depends on durable state.
Complete source code: ihiteshsharma/agent-tool-idempotency-lab
Clone the verified implementation or use it to compare each step in the experiment.
The ambiguity behind every mutating retry
A caller can observe three broad outcomes:
- the operation failed before its side effect committed;
- the operation committed and returned a response;
- the operation committed, but the response was lost.
The third case looks like a failure to the caller and a success to the system of record. A timeout cannot distinguish it from the first case. RFC 9110 permits automatic retries when semantics are idempotent or the client knows the original request was not applied. A dropped response provides no such proof for an ordinary mutation.
Agent systems make this ambiguity routine. Tool calls cross model runtimes, orchestrators, queues, HTTP clients, databases, and third-party providers. Processes restart and leases expire. At-least-once attempts are often the practical delivery model even when the product language says “run once.”
The durable solution is not exactly-once transport. It is a mutation whose repeated delivery has defined semantics.
The governing model is:
Retry safety requires stable identity, canonical intent, serialized ownership, and a durable replayable result.
Without intent binding, a key can replay the wrong operation. Without serialization, concurrent attempts race. Without a durable result, a retry cannot recover from a lost response.
Retry is a policy decision
Transport libraries often retry on timeout, connection reset, or selected status codes. That mechanism cannot decide whether repeating a business operation is safe. The answer depends on what the operation means and where its side effect commits.
A retry policy should therefore declare the attempt budget, backoff, deadline, and retryable error classes together with the operation's semantic category. A read may tolerate several attempts. A database-local conditional mutation may retry under one stable key. A payment or deployment may require reconciliation before another attempt. An approval-gated action may forbid automatic retry entirely after an ambiguous response.
Backoff protects a struggling dependency but does not protect business state. Jitter reduces synchronized retry storms but does not prevent duplicates. Circuit breakers stop repeated calls during failure but cannot resolve an already ambiguous commit. These reliability mechanisms complement an idempotency contract; none substitutes for it.
For agents, the trusted orchestrator should generate the operation identity from durable run and step state. Asking the model to invent a key makes identity sensitive to prompt variation and model behavior. The model may propose intent, but the execution boundary owns retry semantics.
Typed contracts come before deduplication
Idempotency compares repeated intent, so intent must have a stable shape. A permissive tool that ignores unknown fields, coerces types differently across versions, or applies changing defaults cannot reliably decide whether two attempts mean the same thing.
A typed contract defines required fields, allowed values, normalization, and rejection behavior. Validation happens before hashing or reserving the key. Unknown fields fail rather than silently disappearing from the canonical form. Enumerations prevent semantically different strings from collapsing into one path. Tool version is part of the scope whenever a schema or default changes meaning.
Canonicalization should include values that affect the side effect and exclude transport-only metadata such as trace IDs. It should preserve distinctions the business cares about: tenant, destination, amount, resource version, and authorization context may all belong to intent. The idempotency key itself does not; it identifies the intent rather than describing it.
This yields a useful test: if two requests produce the same canonical hash, replaying the stored outcome must be acceptable to the caller. If that statement is false, the canonical form is missing a meaningful field.
Identity, intent, and result
Safe retry requires three values that are related but not interchangeable.
Operation identity is a stable idempotency key supplied by a trusted caller. Every attempt for the same logical mutation carries the same key.
Canonical intent is the validated request after defaults, normalization, and field rules are applied. It is serialized deterministically and hashed.
Durable result is the resource identifier and response stored with the operation outcome.
The key alone is unsafe. Suppose run-42:create-ticket first means “restore the search index” and later means “delete the search index.” Blindly replaying the first response would hide a caller bug and attach the wrong result to new intent. The stored hash binds identity to meaning. Reusing a key with a different hash becomes a conflict.
Canonicalization is part of the public contract. Changing a default, trimming rule, or schema can change the hash for logically equivalent input. Production systems therefore version the tool contract and canonicalization rules rather than treating serialization as an implementation detail.
A durable operation state machine
An idempotency table is more than a cache. It is a state machine:
- Absent: no attempt has reserved the identity.
- Pending: an attempt owns the identity but has not stored a completed result.
- Completed: the side effect and replayable result committed.
- Conflicted: the key exists but the new canonical intent differs.
For a database-local side effect, the operation record, business mutation, and result should commit in one transaction. The first attempt reserves the key, creates the resource, stores the result, and commits. An identical retry reads the completed result. A changed request fails. A failure before commit leaves no durable outcome. A response loss after commit leaves the completed outcome available for replay.
The state model also exposes a dangerous case: a durable pending row without an atomic side effect or result. The system needs a recovery rule—owner lease, timeout, reconciliation, or manual review—rather than guessing that the mutation is safe to repeat.
Pending may mean a live owner, failure before the side effect, or an unrecorded external success. A lease distinguishes live from abandoned ownership, but only reconciliation resolves an ambiguous external outcome. Blindly stealing expired work can duplicate the action the table was meant to protect.
Completed results need an error model. A deterministic business rejection may be replayable; a transient transport failure usually is not. Persisting every exception can make an outage permanent, while persisting none can repeat irreversible work. The contract must define which outcomes bind the key.
Concurrent retries need serialization
Sequential demos can conceal the real race. Two workers may read “absent” simultaneously and both attempt the side effect. Correctness must come from a shared authority: a uniqueness constraint, conditional write, or transaction that serializes reservation.
The lab uses SQLite BEGIN IMMEDIATE. SQLite documents that an immediate transaction starts the write transaction at once, and SQLite permits only one simultaneous writer. That makes the read-reserve-create sequence easy to reason about, though it is also a throughput ceiling. A production database may use a unique constraint plus an insert-on-conflict path, but it must preserve the same state transition.
Process-local locks and in-memory maps are insufficient. They disappear on restart, do not coordinate multiple replicas, and cannot resolve an outcome committed by another process.
The transaction boundary is the real boundary
The SQLite pattern works because the operation record and ticket share one atomic database transaction. It does not make an email, cloud deployment, payment, or remote ticket-provider call atomic with the local row.
For an external side effect, choose among:
- propagate the same idempotency key to a provider that supports it;
- write an outbox event transactionally, deliver it at least once, and make the consumer idempotent;
- assign the final resource identity before delivery so repeated creation converges;
- record the ambiguous attempt and reconcile against the provider before retrying.
These mechanisms move or coordinate the authority. None creates a magical transaction across independent systems. The useful production goal is narrower: prevent duplicates where possible, detect them otherwise, and make ambiguous outcomes recoverable.
A uniqueness constraint prevents two rows from claiming the same identity, but does not by itself preserve the original result or detect changed intent. A durable idempotency record adds that binding and replay contract. A provider idempotency key delegates the same boundary to the system that owns the remote side effect; its retention and replay rules remain part of the caller's contract.
Classify a tool before choosing retry policy
Use four categories:
- Read-only: repetition has no business side effect, though cost and rate limits still matter.
- Naturally idempotent: repeating “set status to closed” converges on the same state.
- Conditionally idempotent: mutation is safe only with stable identity, intent binding, and durable replay.
- Non-idempotent external effect: safety requires provider support, an outbox, reconciliation, compensation, or human approval.
This classification belongs in the tool contract. An orchestrator should not infer retry safety from a function name or HTTP method.
Experiment: prove the contract
The repository artifact is tool_lab.py. It uses only Python, SQLite, SHA-256, and canonical JSON. The mutation validates exactly four fields, scopes the key by tenant and tool name, and stores the ticket plus serialized result in one transaction.
git clone https://github.com/ihiteshsharma/agent-tool-idempotency-lab.git
cd agent-tool-idempotency-lab
python3 -m unittest -v test_lab.py
python3 tool_lab.py
The controlled sequence creates ticket 1, replays the same request, reuses the key with a changed title, then creates ticket 2 while injecting a synthetic connection loss after commit. The final retry must recover ticket 2 from the operation table.
Verification and observed behavior
The verification gate is semantic: equivalent intent must return the same resource, changed intent must fail, retry after an ambiguous commit must not increase the business-row count, and the result must survive the producing process.
The executed run on CPython 3.14.6 and SQLite 3.53.2 returned verdict: passed: two operation rows, two ticket rows, and one durable side effect per logical operation. Equivalent replay returned the original identifier. Changed intent raised IntentConflict. The lost-after-commit retry returned the committed result without a third ticket.
These observations support the key claim inside one SQLite transaction boundary: equivalent and ambiguous retries replay one durable result, while changed intent fails closed. They do not prove coordination across databases, regions, or independently committed remote effects.
Failure, recovery, and interpretation
The changed-intent test is a safety failure: recovery is to reject the request and require a new key for genuinely new intent. The lost-response test is an availability failure: recovery is to replay the stored result.
If an equivalent retry creates another ticket, reservation or transaction boundaries are wrong. If changed intent receives the old result, canonicalization or conflict handling is wrong. If recovery depends on a running Python process, durability is missing. If the local invariant passes but an external provider duplicates work, the authority is placed at the wrong boundary.
Observability and security
Record tenant scope, tool version, operation name, key, request-hash prefix, state transition, disposition, resource ID, transaction latency, conflict count, replay count, and recovery age. Do not log raw arguments by default; tool bodies may contain credentials, payment data, or private documents.
Idempotency is not authorization. An authenticated tenant must not retrieve another tenant's result by guessing a key. Scope uniqueness to tenant and tool version, encrypt sensitive stored results, rate-limit new keys, set retention to the maximum retry/reconciliation window, and audit reads of consequential outcomes.
Production implications and productionization gap
SQLite's single-writer behavior is appropriate for the laptop proof, not a universal service design. Production adds multi-process concurrency tests, busy/lock policy, schema and canonicalization migrations, expiry, quotas, stuck-pending alerts, provider reconciliation, disaster recovery, and contract compatibility tests.
Workflow checkpoints and provider idempotency keys are alternatives only when they own the correct boundary. They do not remove the need to define stable identity, changed-intent behavior, and durable replay.
A review checklist for every mutating tool should ask:
- Who creates the stable key, and for how long is it valid?
- Which validated fields define canonical intent?
- What authority serializes concurrent attempts?
- Which result and error classes are replayable?
- Can the side effect and operation record commit atomically?
- If not, how is the external outcome reconciled?
- What happens to abandoned pending operations?
- Which metrics reveal conflicts, replay storms, and stuck recovery?
If these questions have no explicit answers, “retry enabled” is an undocumented correctness policy.
Retries are delivery behavior; idempotency is a business-semantic guarantee.
Cleanup
The lab uses a temporary database and deletes it in finally. It creates no cloud resource and needs no credentials.