Distributed Platforms 3 Aug 2026 6 min read

The KV Cache Is Your Real LLM Serving Budget

Derive KV-cache memory from model architecture, then connect retained tokens and active sequences to real serving capacity.

A quantized model can fit in memory and still fail to serve the intended workload. The weights are shared across requests, but every active sequence accumulates its own attention state. Longer prompts, longer conversations, and more concurrent sessions spend that state budget even though the model file never changes.

That growing state is the key-value cache. It is why an advertised context window is not free capacity and why “the model uses 2 GB” is not yet a serving plan.

To reason about capacity, start with why autoregressive decoding retains state at all.

Complete source code: ihiteshsharma/kv-cache-serving-budget
Clone the verified implementation or use it to compare each step in the experiment.

Why autoregressive decoding needs a cache

A decoder-only transformer generates one token at a time. During prefill it processes the known prompt and computes attention keys and values for every layer. During decode, each new token attends to the retained sequence.

Without caching, the runtime would recompute the keys and values for all earlier tokens at every generation step. Keeping them exchanges repeated computation for memory capacity and memory traffic.

For every retained token and transformer layer, the runtime stores:

key vectors for each KV head;
value vectors for each KV head.

Each new token adds another position. Each unrelated sequence owns its logical history. Model weights can be shared; arbitrary request histories cannot.

This gives the first mental model:

Context and concurrency consume per-request state that is separate from shared model weights.

Derive the first-order formula

For a conventional full-attention decoder, a useful estimate is:

KV bytes = layers
           × 2                    # key and value
           × KV heads
           × head dimension
           × bytes per element
           × retained tokens
           × active sequences

The architecture contributes layers, KV heads, head dimension, and cache precision. The workload contributes retained tokens and active sequences.

The estimate excludes model weights, allocator metadata, block tables, temporary tensors, runtime workspaces, request objects, and operating-system accounting. It is not total process memory. Its value is that it makes the architecture-workload relationship explicit.

Query heads and KV heads are not always equal

In multi-head attention, each query head has its own key and value head. Grouped-query attention shares a KV head across a group of query heads. Multi-query attention shares one KV head across all query heads.

The formula therefore uses KV heads, not the number of query heads. Two models with similar parameter counts and maximum contexts can have very different per-token state costs.

This is one reason to read pinned architecture metadata rather than infer capacity from a marketing name or parameter count.

Work one configuration by hand

Use a Qwen2.5-3B-style teaching configuration:

{
  "layers": 36,
  "kv_heads": 2,
  "head_dim": 128,
  "bytes_per_element": 2
}

For f16 keys and values, one retained token costs:

36 × 2 × 2 × 128 × 2 = 36,864 bytes

That is exactly 36 KiB per token.

One 4096-token sequence requires:

36 KiB × 4096 = 144 MiB

Four independent sequences require:

144 MiB × 4 = 576 MiB

Moving from one sequence to four adds 432 MiB of first-order KV state. The arithmetic is simple; the important step is mapping it to the runtime's allocation controls.

Context is an allocation, not only a capability

A model may support a large maximum context, but a server still decides how much state to reserve or admit for active work.

Four 512-token conversations retain roughly one eighth of the worst-case state of four 4096-token conversations. One long RAG request can occupy capacity that might otherwise admit several short requests. Request count alone is therefore a weak admission signal.

A scheduler needs retained tokens, expected output growth, available cache blocks, latency objectives, and a policy for queueing, preemption, or rejection.

When comparing configurations, preserve the quantity you intend to test. A control with one 4096-token slot and an expanded server with four slots should allocate a total context of 16384 if the purpose is to preserve 4096 tokens per slot. Changing slots while shrinking per-slot context would change two variables at once.

Cache precision is a separate design choice

The formula contains bytes per element, so a lower-precision K/V representation can reduce capacity pressure. It is not automatically a free improvement.

Runtime support, kernel paths, model architecture, device, and task sensitivity determine whether a lower-precision cache improves useful capacity. Weight quantization and KV-cache quantization are also separate decisions: a Q4 GGUF can still use f16 K and V state.

For every cache type, evaluate admitted tokens and sequences, TTFT, TPOT, long-context task quality, preemption, recomputation, process or device memory, and cost per good request. Reporting only the model filename does not describe live cache precision.

Paging improves allocation, not the tensor requirement

Requests grow and finish at different times. Reserving one large contiguous region per request can waste memory through over-allocation and fragmentation.

PagedAttention uses an operating-system-inspired paging model so a logical sequence does not require one contiguous physical region. Block tables and partially filled pages add overhead, while prefix sharing or copy-on-write can reduce duplication in suitable workloads.

Paging improves how effectively the runtime uses capacity. It does not make the keys and values for retained tokens disappear.

Why process memory will not equal the formula

A process footprint includes far more than logical KV tensors:

model mappings and quantized weights;
runtime and compute buffers;
allocator pages and metadata;
temporary tensors;
threads, stacks, request state, and networking;
shared, private, resident, and compressed pages reported according to operating-system rules.

The formula predicts one component. A process tool measures an implementation and an accounting model. They should be compared without pretending they are the same quantity.

An unexplained difference is a prompt for runtime-specific measurement, not permission to label every residual byte as KV cache.

Capacity still needs a latency objective

Memory determines whether the server can retain work. Compute and scheduling determine whether that work finishes on time.

The existing Apple M2 Pro benchmark illustrates the distinction. A four-slot configuration could complete eight offered requests, yet only four of eight met both TTFT and TPOT objectives. Availability looked healthy while goodput exposed the actual serving boundary.

A capacity plan therefore joins cache occupancy to TTFT, TPOT, end-to-end latency, queueing, preemption, and goodput. “It fits” is only the first gate.

A decision framework for KV capacity

Before deploying, answer:

What are the model's layer, KV-head, head-dimension, and cache-precision values?
How many tokens does the workload retain per active sequence?
How many sequences must be active concurrently?
How does the runtime page, share, evict, offload, or quantize cache state?
What process or device headroom remains at peak load?
Which admitted workload still meets latency and quality objectives?

If these values are not visible, the statement “the model fits” is not a capacity plan.

Experiment: calculate the cache and compare hardware evidence

The standalone lab encodes the formula and the pinned dimensions, then compares the predicted cache growth with existing whole-process observations from a CPU-serving experiment.

That experiment used an Apple M2 Pro with 16 GiB unified memory, llama.cpp release b10217 at commit ddd4ec142, Qwen2.5 3B Instruct Q4_K_M, four CPU threads, zero GPU layers, f16 K/V defaults, and approximate warm vmmap -summary observations.

Run the calculator:

git clone https://github.com/ihiteshsharma/kv-cache-serving-budget.git
cd kv-cache-serving-budget
python3 -m unittest -v test_lab.py
python3 kv_lab.py

The lab requires no model download or accelerator. It calculates the analytical values and keeps the recorded process observations in a separate comparison layer.

Verification and observed behavior

The fresh run executed three tests and returned "verdict": "passed".

Configuration	Derived KV state	Approximate whole-process footprint
1 × 4096 tokens	144 MiB	2.0 GiB
4 × 4096 tokens	576 MiB	2.5 GiB
Change	432 MiB	512 MiB

The calculator reports an 80 MiB residual between the predicted KV change and approximate process change. It also rejects zero or negative cache dimensions rather than producing a confident but meaningless estimate.

What the comparison establishes

For the pinned architecture and f16 cache precision, KV state scales linearly from 144 MiB for one 4096-token sequence to 576 MiB for four. The predicted 432 MiB increase is consistent with KV growth being a substantial part of the approximate 512 MiB process change.

It does not prove that the residual is another KV tensor, nor does it predict total serving memory exactly. Process memory contains additional state, and the observations are rounded.

The lab also does not establish allocator efficiency, latency, throughput, sliding-window behavior, prefix-sharing benefit, model quality under cache quantization, or a universal concurrency limit.

Observability, security, and production implications

Useful cache telemetry includes retained tokens, capacity, occupancy, allocation failures, fragmentation, prefix hits and evictions, preemption, swap, recomputation, active and queued requests, TTFT, TPOT, goodput, cache precision, and runtime revision.

KV state is derived from prompts, generated text, retrieved documents, tool schemas, and conversation history. Treat it as sensitive even though it is not human-readable text. Tenant isolation, cache invalidation, crash artifacts, debugging dumps, and prefix sharing belong in the threat model.

A production study should read pinned model metadata, verify the runtime's allocation behavior, measure the target hardware, replay real prompt and output distributions, include safety headroom, and join memory limits to latency and quality gates.

Cleanup

The calculator writes no persistent state. For a live runtime experiment, stop the server, confirm the listener is gone, and retain only reviewed, non-sensitive evidence.