Benchmark LLM Serving Without Lying to Yourself
Understand inference performance as a workload-dependent surface, then test saturation and goodput with a controlled local benchmark.
Read deep diveStaff engineer · AI platforms · Mumbai
Useful intelligence is an engineering problem.
Engineering notes on agentic systems, retrieval, distributed platforms, and the work between prototype and production.
Agentic AIRAGSearchDistributed systems
The interesting work begins after the demo works.
Start here
Featured note
Latest notes
Recent engineering notes
Evaluation Is the Control System for Production AI - Production AI Systems - Part 5
Evaluation connects product requirements, versioned datasets, component failures, operational constraints, release gates, and production feedback.
Read noteLong-Running AI Work Is a Durable Workflow - Production AI Systems - Part 4
Long-running AI work should be modeled as durable state transitions with idempotent stages, checkpoints, bounded retries, cancellation, and backpressure.
Read noteFrom Model Output to Trusted Application Data - Production AI Systems - Part 3
Schema-constrained generation is only the first trust boundary. Production systems still need evidence, policy, authorization, bounded repair, and explicit disposition.
Read noteThe Model Gateway Is a Policy Boundary, Not a Proxy -Production AI Systems - Part 2
A model gateway should centralize provider compatibility, routing, quotas, and data policy without absorbing product workflow logic.
Read noteA Production AI System Is More Than a Model Call - Production AI Systems - Part 1
A production AI feature is not a model call. It is a distributed system with explicit boundaries for authorization, orchestration, validation, durable execution, and evaluation.
Read noteWhen an Agent Calls a Tool, What Happens to Its KV Cache?
Understand how paused agent requests trade accelerator memory against KV-cache transfer, recomputation, resume latency, and system goodput.
Read noteWorking areas