Evaluation Is the Control System for Production AI - Production AI Systems - Part 5
Evaluation connects product requirements, versioned datasets, component failures, operational constraints, release gates, and production feedback.
Read noteThe complete notebook
All articles
Deep dives and field notes on intelligent systems, search, and backend platforms.
Long-Running AI Work Is a Durable Workflow - Production AI Systems - Part 4
Long-running AI work should be modeled as durable state transitions with idempotent stages, checkpoints, bounded retries, cancellation, and backpressure.
Read noteFrom Model Output to Trusted Application Data - Production AI Systems - Part 3
Schema-constrained generation is only the first trust boundary. Production systems still need evidence, policy, authorization, bounded repair, and explicit disposition.
Read noteThe Model Gateway Is a Policy Boundary, Not a Proxy -Production AI Systems - Part 2
A model gateway should centralize provider compatibility, routing, quotas, and data policy without absorbing product workflow logic.
Read noteA Production AI System Is More Than a Model Call - Production AI Systems - Part 1
A production AI feature is not a model call. It is a distributed system with explicit boundaries for authorization, orchestration, validation, durable execution, and evaluation.
Read noteWhen an Agent Calls a Tool, What Happens to Its KV Cache?
Understand how paused agent requests trade accelerator memory against KV-cache transfer, recomputation, resume latency, and system goodput.
Read noteThe KV Cache Is Your Real LLM Serving Budget
Derive KV-cache memory from model architecture, then connect retained tokens and active sequences to real serving capacity.
Read noteEvaluate the Agent Trajectory, Not Just Its Final Answer
Build an execution-based grader that separates final-state success from tool-selection, argument, authorization, ordering, and duplicate-work failures.
Read noteBuild a Lexical Search Baseline Before Adding Embeddings
Use lexical retrieval as the experimental control for judging when dense, hybrid, or reranked search actually improves evidence selection.
Read noteMake Agent Tool Retries Safe With Typed Contracts and Idempotency
Understand idempotency as a semantic contract for ambiguous tool outcomes, then prove durable replay and conflict handling locally.
Read note