When an Agent Calls a Tool, What Happens to Its KV Cache?
Understand how paused agent requests trade accelerator memory against KV-cache transfer, recomputation, resume latency, and system goodput.
Read noteTopic
Distributed Platforms
The KV Cache Is Your Real LLM Serving Budget
Derive KV-cache memory from model architecture, then connect retained tokens and active sequences to real serving capacity.
Read noteWhen Q4 Passes Latency but Fails Quality
See why a Q4 model that fits and meets latency can still fail a release gate—and why incomplete evidence must not select a quantization.
Read noteServe a 3B LLM on CPU With llama.cpp
Understand CPU inference from first principles: autoregressive decode, memory bandwidth, KV state, context allocation, scheduling, and admission control.
Read noteBenchmark LLM Serving Without Lying to Yourself
Understand inference performance as a workload-dependent surface, then test saturation and goodput with a controlled local benchmark.
Read note