From Model Output to Trusted Application Data - Production AI Systems - Part 3
Schema-constrained generation is only the first trust boundary. Production systems still need evidence, policy, authorization, bounded repair, and explicit disposition.
Part 3 of 5 in Production AI Systems. Part 2 defined a stable, policy-aware path to models. This part addresses what happens after generation.
An application requests a structured support-case summary and receives this response:
{
"caseId": "case_123",
"issueSignals": ["Kafka timeout", "ingestion lag"],
"severity": "high",
"confidence": 0.94
}
It is valid JSON. Every field appears plausible. The response can be deserialized into a typed object. None of those facts establishes that the case is high severity, that the Kafka timeout appears in authorized evidence, or that the requesting user may act on the record.
A schema makes generated output structurally usable. It does not make the output true, authorized, or safe.
The output-trust boundary must treat model responses as untrusted input and increase confidence through deterministic checks.
Structure is the first rung, not the finish line
There is a useful reliability ladder for model output:
flowchart LR
A[Free-form text] --> B[Prompted JSON]
B --> C[JSON mode]
C --> D[Tool or function call]
D --> E[Schema-constrained generation]
E --> F[Application validation]
F --> G[Evidence and policy validation]
Each rung removes a class of ambiguity. Prompted JSON may still include prose, renamed fields, or malformed syntax. JSON mode can ensure parsability without ensuring the intended shape. Tool calls make an intended operation explicit. Schema-constrained generation can restrict fields and types during decoding.
Application validation remains necessary at every rung. JSON Schema Draft 2020-12, for example, defines vocabularies for structure and validation. It can express required properties, types, ranges, arrays, and whether additional fields are allowed. It cannot prove that a generated claim is supported by a source document or permitted by business policy.
This is the core distinction: structural validity is about representation; application validity is about meaning and permitted use.
Validate in layers
A single valid: true flag hides too much. Validation should preserve which layer accepted or rejected the output.
1. Syntax
Can the response be parsed under the expected encoding and serialization format? A failure here is often repairable without changing meaning: removing a code fence or extracting the sole JSON object may be safe when the transformation is unambiguous.
2. Structure
Does the object match the declared schema? Required fields, types, enums, ranges, string lengths, array limits, and unknown fields belong here. The schema version must travel with the result because consumers need to know which contract they received.
3. Semantics and evidence
Do referenced entities exist? Are claims supported by authorized source material? Does the cited text actually contain the asserted error signature? Is a calculated incident duration consistent with the timestamps in the case history?
Evidence references are stronger than an unsupported confidence score:
{
"signal": "Kafka timeout",
"evidence": {
"documentId": "attachment_123",
"chunkId": "chunk_18",
"quote": "Kafka requests timed out during ingestion"
}
}
The application can verify that the document exists, belongs to the correct tenant, and contains the referenced text. This still does not establish every interpretation, but it makes important claims inspectable.
4. Business policy
May this result be used automatically? Does the workflow require human review? Are there prohibited fields or decisions? Is the user authorized to request this action for this support case?
Policy validation is deliberately last. A perfectly supported output can still be impermissible for the requested use.
flowchart TD
O[Generated Output] --> P{Parseable?}
P -->|No| R[Bounded Repair]
P -->|Yes| S{Schema Valid?}
S -->|No| R
S -->|Yes| E{Evidence Supported?}
E -->|No| H[Review or Reject]
E -->|Yes| B{Policy Allows Use?}
B -->|No| X[Reject and Audit]
B -->|Yes| A[Accept]
Repair must match the failure
Repair is useful when it is narrow and bounded. It is dangerous when it becomes a loop whose only goal is to produce something that passes.
Deterministic repair is appropriate for representation defects with one obvious correction. A schema failure may justify one regeneration attempt that includes the validation errors. An evidence failure should return to the original evidence, not ask a model to cosmetically rewrite an unsupported claim. A policy failure is not repairable by the model at all.
Every repair policy needs limits on attempts, elapsed time, and cost. It also needs an exhaustion outcome: reject, degrade to a deterministic result, or route to human review. Unlimited regeneration converts a quality problem into an availability and spend problem.
Partial success is a product contract, not an error-handling shortcut. An extraction may return verified fields while marking another field not_found, but only if downstream consumers understand that distinction. Silently replacing missing values with empty strings destroys evidence about incompleteness.
Model confidence is not a decision policy
A model-generated confidence value is another model output. It may be useful as one signal, but it is not automatically calibrated and should not override deterministic failures.
A disposition can combine evidence coverage, schema completeness, retrieval quality, historical task precision, input risk, and validated calibration data. High-risk tasks may require human review regardless of a confidence score. Low-risk tasks may accept a result only when all deterministic checks pass and measured performance supports the threshold.
The threshold must be derived from evaluation data and segmented by task category. One global number can hide that a system is reliable for standard cases and weak for multilingual or incomplete case histories.
Tool calls are proposed side effects
Structured tool calls make intent explicit, but they do not grant permission. A model proposal such as:
{
"tool": "update_case_status",
"arguments": {
"caseId": "case_123",
"status": "escalated"
}
}
must cross the same trust boundary as any external request. The executor validates the arguments, confirms tenant ownership, checks the caller's permission, applies approval policy, and supplies an idempotency key before invoking a side effect.
Current MCP guidance reflects this separation: tool schemas describe callable interfaces, while clients remain responsible for authorization, trust, and human control over risky invocations. Tool metadata itself should not be treated as trusted merely because a model can see it.
The safe sequence is therefore:
sequenceDiagram
participant M as Model
participant V as Schema Validator
participant P as Policy and Authorization
participant I as Idempotency Store
participant T as Tool
M->>V: Proposed tool call
V-->>P: Structurally valid arguments
P->>P: Check user, tenant, risk, approval
P->>I: Reserve idempotency key
I-->>P: New or matching operation
P->>T: Execute authorized call
T-->>P: Result
P->>I: Persist outcome
Version the complete output contract
A safe schema change requires more than editing a type definition. Prompt behavior, model behavior, validators, storage, and consumers may all depend on the version.
A controlled rollout introduces the new schema, runs offline evaluation, generates old and new forms in shadow where practical, updates consumers, promotes the new version, and preserves rollback. Every stored result retains its schema and prompt versions.
Backward compatibility is a product decision. A compatibility layer can translate additive changes, but it should not manufacture missing evidence or reinterpret semantics silently. When meaning changes, a new contract is safer than a clever adapter.
Observe decisions, not just parse errors
The useful operational unit is the disposition of an attempted result. A trace should show the prompt, model, schema, and validator versions; which layer failed; whether repair occurred; how many attempts were consumed; and whether the final result was accepted, reviewed, or rejected.
Aggregate metrics should include first-pass schema validity, evidence-failure rate, repair success, human-review rate, attempts per accepted result, and cost and latency per accepted result. Segment them by task, language, input category, model, and schema version. A global 98 percent validity rate may conceal that one document type fails half the time.
Logs require the same trust discipline as outputs. Structured payloads may contain personal data, proprietary documents, or credentials returned by tools. Preserve identifiers and decisions needed for diagnosis while applying redaction, access controls, and retention limits to raw content. Observability that creates a second uncontrolled copy of sensitive context is not a reliability improvement.
Design packet: output disposition matrix
| Validation outcome | Default disposition | Reason |
|---|---|---|
| Unambiguous formatting defect | Deterministic repair | Meaning does not change |
| Schema violation | Regenerate within a bounded budget | Model may satisfy the declared contract |
| Missing or unverifiable evidence | Reject or human review | Formatting cannot create support |
| Authorization or policy violation | Reject and audit | Model cannot grant permission |
| High-risk valid result | Human review | Consequence exceeds automation threshold |
| Valid, supported, and permitted | Accept and persist provenance | All required boundaries passed |
| Budget exhausted | Fail explicitly | Hidden loops are worse than a visible failure |
The matrix should be decided before launch. Otherwise, error handling becomes an accidental collection of retries added after incidents.
The support-case-summary workflow now has a trustworthy output boundary, but parsing, retrieval, generation, repair, and review may take longer than one request can safely remain open. Part 4 moves the workflow into durable execution.