Evaluation Is the Control System for Production AI - Production AI Systems - Part 5
Evaluation connects product requirements, versioned datasets, component failures, operational constraints, release gates, and production feedback.
Part 5 of 5 in Production AI Systems. Part 4 turned multi-stage AI work into a durable workflow. The final boundary governs how that system changes.
A team changes the support-case-summary prompt and compares three examples by hand. The new summaries sound clearer, so the change ships. A week later, correction rates rise for multilingual customer messages, average cost increases because outputs are longer, and a previously fixed unsupported-claim failure returns.
Nothing in the deployment failed. The change passed ordinary software tests. The model returned valid responses. The release process still had no evidence that the system improved.
Evaluation is not a final benchmark. It is the control system for production AI development.
It connects product requirements, versioned inputs, system traces, graders, operational constraints, and release decisions. Without that loop, prompt and model changes remain informed guesses.
Define the task before selecting metrics
“Generate a good support-case summary” is not an evaluable product requirement. A stronger task definition might be:
Given an authorized support case and service policy, produce a concise summary of verified symptoms, relevant context, attempted fixes, and missing evidence. Support consequential claims with source evidence and make no claims absent from the supplied records.
This definition creates observable dimensions:
- authorization correctness;
- schema validity;
- factual and evidence support;
- relevance to the role;
- completeness of material facts and gaps;
- concision;
- safety and policy compliance;
- latency and cost.
The metric follows the requirement. A schema-validity percentage cannot answer whether claims are supported. A semantic-similarity score cannot prove that unauthorized evidence was excluded. User acceptance is useful product feedback, but it does not establish correctness when users may not notice subtle errors.
Risk also changes the release bar. A draft summary reviewed by a support engineer may tolerate different uncertainty from an automatic escalation or resolution decision. The evaluation plan must therefore record intended use, consequence, review path, and critical failures that are never averaged away.
Evaluate the system at its boundaries
The series architecture provides natural evaluation layers:
flowchart TB
S[Support-Case Summary System]
S --> I[Input and Parsing]
S --> R[Authorized Retrieval]
S --> G[Gateway and Generation]
S --> V[Validation and Policy]
S --> W[Durable Workflow]
S --> O[Operational Outcome]
Each layer answers a different question.
Input evaluation checks whether the source was parsed correctly and whether tables, dates, and sections survived extraction. Retrieval evaluation checks that relevant evidence appeared and unauthorized evidence did not. Generation evaluation checks supportedness, completeness, and relevance. Validation evaluation checks that intended failures were rejected or reviewed. Workflow evaluation checks retries, tool use, termination, idempotency, and final state. Operational evaluation adds latency, throughput, cost, timeout, retry, and fallback behavior.
End-to-end success remains essential, but component scores make failure actionable. When a summary omits Kafka experience, the fix depends on whether parsing lost the sentence, retrieval missed it, context construction truncated it, or generation ignored available evidence.
The dataset is a versioned product asset
An evaluation dataset should represent the product distribution and its risks, not merely examples that are convenient to grade.
For support-case summaries, useful categories include standard cases, intermittent failures, missing timestamps, conflicting evidence, long logs, multilingual messages, unusual attachments, prompt injection in source text, and cases with insufficient diagnostic context. Historical production failures become regression cases after review and redaction.
Each case should carry stable identity, input references, category and risk labels, required facts, forbidden claims, expected evidence, and the grading rubric version. Split documents at the source or template level where leakage between nearly identical documents would make the held-out result optimistic.
Keep at least one holdout set outside routine prompt iteration. If every failure is repeatedly inspected and tuned against, the suite becomes development data rather than independent release evidence.
Ownership matters. Product and domain teams define meaningful tasks and rubrics; platform teams provide repeatable execution, provenance, and comparison. An evaluation platform cannot infer what “good” means from model output alone.
Combine graders according to the question
No single grader covers the system.
Deterministic graders should handle deterministic properties: schema conformance, exact fields, citation existence, permission correctness, tool-argument constraints, state transitions, latency budgets, and cost ceilings. They are reproducible and easy to diagnose.
Model-based graders are useful for residual semantic questions such as relevance, unsupported synthesis, or rubric-based writing quality. They introduce their own variation and bias, so judge prompts, models, and versions belong in provenance. Pairwise comparisons can reduce the difficulty of absolute scoring, but order should be randomized and ties allowed.
Human evaluation remains necessary for calibrating subjective rubrics, adjudicating disagreement, reviewing high-risk failures, and validating model-based judges. Reviewers need the input, authorized evidence, output, rubric, and failure categories - not only a generated answer in isolation.
Anthropic's current evaluation guidance describes a similar layered view for agents: tasks, trials, graders, transcripts, outcomes, and a harness. The important architectural point is that the evaluated object is the model plus the surrounding harness and environment. The same model can behave differently under another prompt, tool set, retrieval configuration, or retry policy.
Segment failures before aggregating scores
One aggregate score makes different failure classes interchangeable. A minor concision regression can offset a critical authorization failure mathematically even though the product should never accept that trade.
Use a failure taxonomy tied to system owners:
INPUT_PARSE_ERROR
UNAUTHORIZED_CONTEXT
RETRIEVAL_MISS
CONTEXT_TRUNCATION
SCHEMA_INVALID
UNSUPPORTED_CLAIM
INCOMPLETE_SUMMARY
POLICY_VIOLATION
WORKFLOW_TIMEOUT
DUPLICATE_SIDE_EFFECT
COST_BUDGET_EXCEEDED
Report performance by category, risk, language, document type, and workflow version. Preserve per-case artifacts so a score change can be inspected. Statistical summaries support decisions; they do not replace failure analysis.
Non-deterministic systems also require repeated trials when variance affects the product. A single pass can establish that one run succeeded, not that the task is reliable. The number of repetitions should follow the consequence and expected variability rather than a universal convention.
Turn evaluation into a release policy
An evaluation run becomes operationally useful when it produces a decision under a declared policy.
release_policy:
quality:
minimum_supported_claim_rate: 0.98
maximum_category_regression: 0.03
operations:
p95_latency_ms_max: 5000
cost_increase_max: 0.15
safety:
unauthorized_context_allowed: 0
critical_policy_failures_allowed: 0
The numbers above are illustrative, not universal recommendations. A real policy must be derived from product impact, baseline evidence, and risk tolerance.
The release record identifies the proposed change and every relevant version: application, workflow, prompt, schema, model route, retrieval configuration, dataset, grader, and judge. It compares the proposed version with the current production configuration, reports segmented regressions, and records the disposition: reject, revise, shadow, canary, or promote.
Offline evaluation is necessary but incomplete. Shadow execution tests the proposed version against representative traffic without affecting users. A canary exposes a controlled segment while monitoring quality proxies, validation failures, latency, cost, and rollback triggers. High-impact workflows may require explicit approval before promotion.
Production completes the feedback loop
Input distributions change. Document templates evolve. Provider behavior changes. New languages and edge cases appear. Even without a source-code deployment, production quality can drift.
Useful signals include correction size, regeneration, human escalation, abandonment, downstream errors, validation failures, retrieval scores, input length, queue time, and fallback frequency. These are indicators, not automatic labels. A user accepting a summary may mean it was correct, or merely that the user did not inspect it.
Reviewed failures should flow back into the dataset with category, risk, expected behavior, and provenance. The loop is controlled:
flowchart LR
C[Proposed Change] --> E[Offline Evaluation]
E --> P{Release Policy}
P -->|Fail| C
P -->|Pass| S[Shadow]
S --> K[Canary]
K --> R[Production]
R --> F[Reviewed Failures]
F --> D[Versioned Dataset]
D --> E
This is the safe form of a self-improving system. Production does not modify itself. Runs generate evidence; people and controlled automation classify failures, propose changes, evaluate them against held-out cases, approve a canary, and preserve rollback.
Design packet: the evaluation release record
Every material AI-system change should answer:
| Release question | Required evidence |
|---|---|
| What changed? | Versioned prompt, model route, retrieval, schema, workflow, or code diff |
| What task should improve? | Explicit outcome and risk definition |
| Against what cases? | Dataset version, categories, holdout policy |
| How was it graded? | Deterministic, model, and human grader versions |
| What improved or regressed? | Per-category metrics and inspected failures |
| At what operational cost? | Latency, tokens, spend, retries, fallbacks |
| Which failures are blocking? | Critical safety and policy gates |
| How will production exposure grow? | Shadow, canary, approval, and rollback plan |
| Can the result be reproduced? | Complete run provenance and retained artifacts |
The consolidated readiness check for the full series is short:
- Authorization filters data before model context is built.
- Workflow ownership is separate from provider integration.
- Generated output crosses structural, evidence, and policy validation.
- Long work has durable state, idempotent stages, and bounded retries.
- Every material change faces versioned evaluation and a rollback-capable release policy.
The model remains important, but it is no longer asked to be the authorization system, workflow engine, validator, queue, or release manager. That is the central architecture of a production AI system: probabilistic computation inside deterministic boundaries, with evidence governing every change.