Serve a 3B LLM on CPU With llama.cpp
Understand CPU inference from first principles: autoregressive decode, memory bandwidth, KV state, context allocation, scheduling, and admission control.
CPU inference is often framed as a cost-saving exercise: choose a small quantized model, confirm that it fits in memory, and expose an API. That framing misses the engineering question. A model that loads successfully may still provide poor interactive latency, collapse under a second user, or reserve so much context state that the host has no operational headroom.
The useful question is narrower: for which workload is this CPU host a valid serving tier? Answering it requires understanding how autoregressive generation consumes memory bandwidth, how KV cache turns context into per-request state, how parallel slots share the model but compete for resources, and how those mechanisms appear in TTFT, TPOT, and goodput.
Complete source code: ihiteshsharma/llm-inference-real-hardware
The repository contains the localhost launcher, streaming benchmark, failure test, and recorded evidence.
Fitting a model is not serving it
A GGUF file describes stored weights and metadata. Quantization reduces the weight representation, which can make a model small enough to map into memory. Serving introduces additional demands: runtime workspaces, tokenization, output buffers, the server process, and attention state for every active sequence. The operating system and neighboring processes still need headroom.
Capacity is not only about bytes. During generation, the runtime repeatedly applies the same model layers to produce one new token. At small batch sizes, relatively little arithmetic is performed for each byte of weight data moved through the memory hierarchy. Decode can therefore become constrained by memory bandwidth before the CPU exhausts its theoretical arithmetic throughput.
This explains why adding threads is not a monotonic optimization. Threads can accelerate work while unused cores and bandwidth remain, then stop helping as memory channels saturate, caches contend, or scheduling overhead grows. A CPU with more cores is not automatically a faster decode engine; model representation, memory subsystem, backend kernels, and workload shape all participate.
The governing model is:
CPU serving allocates a shared model working set, memory bandwidth, compute time, and per-request state under a latency objective.
Context becomes a concurrency budget
A decoder-only model generates text autoregressively. It first processes the known prompt in a prefill phase. Because all prompt tokens are available, much of this work can be parallelized. Prefill also computes the key and value tensors used by attention.
The decode phase then produces one new token per active sequence at each step. Recomputing attention state for every preceding token would be wasteful, so the runtime retains those key and value tensors in a KV cache. Each new token adds more state and reads the state accumulated so far. KV caching exchanges repeated computation for memory capacity and memory traffic.
That exchange turns context length into an operational allocation. KV state grows with retained tokens, model architecture, cache precision, and active sequences. A long advertised context window is therefore not free capability. It consumes part of the state budget that could otherwise admit another request.
Parallel slots share one loaded copy of the model weights, but they do not share one sequence history. Each slot needs its own logical context state. Four short chats and four near-limit RAG prompts have the same slot count but very different memory and prefill demands. Production admission control must reason about tokens and active state, not only request count.
Decide whether CPU is the right tier
Use four gates:
- Quality: does the bounded model pass task-specific held-out checks?
- Memory: do weights, runtime workspace, active context, and host headroom fit without pressure?
- Latency: does the intended prompt/output distribution meet TTFT, TPOT, completion, and error objectives at expected load?
- Operations: can the service be secured, supervised, observed, upgraded, and rolled back economically?
These gates are independent. A fast model that fails the task is not deployable. A model that fits but queues every second request is not an interactive service. A cheap local process without authentication, resource limits, or artifact controls is not production-ready.
CPU is most credible for development, offline or edge operation, background processing, and bounded low-volume endpoints. It becomes a poor tier when quality requires a larger model or the workload requires long prompts, high concurrency, or strict latency.
Prerequisites and implementation budget
Allow 45–75 minutes after downloading the runtime and model. The lab needs Python 3.11 or newer, a pinned llama-server build, and a checksummed Qwen2.5 3B Instruct Q4_K_M GGUF. Record CPU architecture, logical cores, memory, operating system, power state, runtime commit and build flags, model revision and hash, thread count, total context, and parallel slots.
The launcher binds only to 127.0.0.1, enables server metrics, assigns a stable model alias, and defaults to zero GPU layers. The benchmark uses the OpenAI-compatible streaming endpoint and writes one JSONL record per request.
Mental model: three memory budgets
Three budgets explain most first-order serving behavior.
Weight budget. Quantized parameters form the shared model working set. They dominate artifact size and much of the repeated data movement during decode. Changing quantization changes storage, kernels, performance, and potentially output quality.
Runtime budget. The server needs metadata, compute buffers, allocators, tokenization, request objects, and operating-system headroom. This is why file size cannot predict process footprint exactly.
Sequence-state budget. KV cache and request buffers grow with active tokens and sequences. Context and slots spend this budget together. A server can have ample room for weights and still reject or delay work because active sequence state has consumed its useful envelope.
The controls follow from the budgets: quantization and backend influence the weight path; threads influence how compute and bandwidth are used; context bounds sequence state; slots determine how many sequences may compete; admission control limits offered work before tail latency collapses.
Pin the artifact and runtime
An inference deployment is more than a model name. The same friendly name can resolve to a new repository revision, quantization recipe, chat template, or runtime kernel. Any of those changes can alter memory, latency, or behavior.
Record a reproducible identity:
model repository and immutable revision
resolved GGUF filename, byte size, and SHA-256
llama.cpp release or commit
compiler, architecture, and build flags
chat template and sampling controls
Q4_K_M is a recipe, not a quality guarantee. It is a sensible laptop-sized candidate because it reduces the shared weight budget, but the deployment decision still requires the separate quality gate developed in the next article.
Experiment: establish the capacity envelope
Start with one slot and a bounded context. This configuration tests the complete serving path without mixing model execution with slot scheduling:
git clone https://github.com/ihiteshsharma/llm-inference-real-hardware.git
cd llm-inference-real-hardware
python3 -m unittest -v test_lab.py
LLAMA_SERVER=/absolute/path/to/llama-server \
MODEL=/absolute/path/to/qwen2.5-3b-instruct-q4_k_m.gguf \
CTX_SIZE=4096 PARALLEL=1 THREADS=4 GPU_LAYERS=0 \
sh ./serve-local.sh
Readiness is a state transition, not proof that a process exists. Check /health and /v1/models, then send a deterministic streamed completion. The client must parse every event, observe non-empty first content, retain terminal usage, and account for every submitted request.
Establish the one-user baseline
The baseline answers three questions. Can the model complete the intended request? How long does the caller wait for the first token? Once generation begins, how quickly do tokens arrive?
TTFT covers queueing, prompt processing, and initial decode from the caller's perspective. TPOT approximates decode cadence after the first token. End-to-end latency combines both phases and output length. Preserve all three: one aggregate cannot identify whether a regression came from prompt work, waiting, generation, or a longer answer.
Separate warmup from measurement and preserve raw request records. A percentile without coverage and failures can make a partial outage look fast.
Add concurrency without hiding the budget
Slots test whether sharing the loaded weights across active sequences increases useful work. Preserve context per slot while changing slot count:
CTX_SIZE=16384 PARALLEL=4 THREADS=4 GPU_LAYERS=0 sh ./serve-local.sh
In this llama.cpp configuration, total --ctx-size is shared across parallel sequences. Four slots therefore need total context 16384 to preserve 4096 tokens per slot. Using 8192 would reduce each slot to 2048 and change both concurrency and per-request capacity.
More slots can improve aggregate throughput because multiple sequences make progress while sharing the model. They also add KV state and competition for CPU and bandwidth. The production question is not “did tokens per second increase?” It is “did more requests complete inside their latency contract?” That quantity is goodput.
Verification and observed behavior
The reference run used llama.cpp b10217 at commit ddd4ec142, Qwen2.5-3B-Instruct Q4_K_M with SHA-256 626b4a6678b86442240e33df819e00132d3ba7dddfe1cdc4fbb18e0a9615c62d, an Apple M2 Pro with 16 GiB unified memory, four CPU threads, and zero GPU layers.
Warm process footprint was measured with macOS vmmap -summary. The control occupied 2.0G; the expanded configuration occupied 2.5G.
| Envelope | Slots × context | Footprint | Highest passing concurrency | First failing concurrency |
|---|---|---|---|---|
| Control | 1 × 4096 | 2.0G | 2 | 4 |
| Expanded | 4 × 4096 | 2.5G | 4 | 8 |
The gates were TTFT p95 at or below 1500 ms, TPOT p95 at or below 120 ms, no failed requests, and goodput equal to submitted requests. At expanded concurrency 4, all eight requests passed: TTFT p95 was 515.831 ms, TPOT p95 was 62.793 ms, and end-to-end p95 was 3719.724 ms.
The bounded result matches the mental model: extra slots increased useful concurrency and footprint but did not create unlimited capacity. Active sequences still shared finite resources.
Failure injection and recovery
The negative test kept the four-slot server fixed and offered concurrency 8. Every request completed, but only four of eight met both latency gates; TTFT p95 reached 3404.986 ms. Availability alone would have missed this failure.
Recovery reduced offered concurrency to 4 without changing the model or server allocation. Goodput returned to 8/8, with TTFT p95 133.118 ms, TPOT p95 54.403 ms, and end-to-end p95 2985.832 ms.
This is admission control in miniature. Once offered work exceeds the useful envelope, accepting everything converts overload into user-visible waiting. Reducing or rejecting work can preserve the contract for admitted requests.
Observability and debugging
Metrics should map back to mechanisms:
- rising TTFT with stable TPOT points toward waiting or prefill pressure;
- rising TPOT points toward decode contention, memory bandwidth, thread placement, or host competition;
- active and deferred requests expose scheduler pressure;
- context occupancy represents KV-state commitment;
- goodput reveals whether completed work still satisfies the service objective.
Collect startup logs, model identity, prompt and predicted token counters, active/deferred requests, physical footprint, CPU use, and termination reason. Change one control at a time; otherwise the result cannot identify whether context, slots, threads, or offered load caused the movement.
Security and isolation
OpenAI-compatible syntax is not a security boundary. The lab has no production authentication, tenant authorization, TLS, quotas, or prompt redaction and must remain on localhost. Treat model files as supply-chain inputs: verify publisher, revision, checksum, and license.
Remote exposure needs authenticated ingress, network policy, request-size and concurrency limits, log redaction, resource isolation, and an explicit retention policy for prompts and outputs.
Cost, scaling, and alternatives
CPU inference exchanges accelerator reservation for CPU time, ordinary memory capacity and bandwidth, higher latency, and lower concurrency. Compare cost per good request, not hardware price or generated token alone.
Ollama and desktop applications reduce setup effort. Direct llama.cpp exposes the capacity controls and metrics needed here. vLLM is a stronger fit when GPU-oriented continuous batching and higher throughput dominate. A hosted API removes serving operations but changes privacy, availability, cost, and control boundaries.
Scale out or change tiers when representative prompts no longer fit the context policy, admission rejects too much work, TPOT violates the interaction target, the required model fails the CPU budget, or operational cost exceeds an accelerator or hosted alternative.
Decision record
On the recorded machine and short-prompt workload, one 4096-token slot supported offered concurrency 2; four 4096-token slots supported concurrency 4. The next tested loads failed because TTFT crossed the gate, not because requests crashed or TPOT crossed its limit.
That conclusion is a workload envelope, not a host specification. Longer RAG prompts, more output tokens, another quantization, different hardware, or stricter latency requires a new measurement.
Serving feasibility is also not release approval. The Q4 artifact failed the separate fixed quality floor in the next experiment. Production selection requires both systems performance and task quality.
Productionization gap
A local process does not provide supervision, distinct liveness and readiness behavior, authenticated TLS ingress, artifact promotion, resource limits, canary rollout, admission control, capacity alerts, failure-domain planning, or rollback.
Production capacity testing needs real prompt/output distributions, open-loop arrival rates, multi-tenant isolation, cold and warm paths, cache behavior, cancellations, and cost per good request. The local lab supplies the mental model and measurement contract; it does not substitute for traffic-shaped validation.
Cleanup
Terminate llama-server and confirm the listener and process are gone. Retain reviewed evidence, then remove raw JSONL that may contain prompts. Delete the GGUF only when it is no longer needed, recording the checksummed artifact removed.