Benchmark LLM Serving Without Lying to Yourself

An LLM server can produce an impressive tokens-per-second number and still deliver a poor product. A warm, single-request benchmark hides the delay before the first token, uneven streaming, queue buildup, failed requests, and the point at which additional load stops producing useful work.

The outcome is a repeatable local benchmark for an OpenAI-compatible server. It records TTFT, TPOT, end-to-end latency, failures, and goodput, then deliberately overloads the server to find the last load level meeting the service objective.

Complete source code: ihiteshsharma/llm-inference-real-hardware
Clone the verified harness or use it to compare each step in the experiment.

Speed is a surface, not a scalar

“How many tokens per second?” sounds like a capacity question, but it omits the coordinates needed to answer it. A serving result belongs to a surface whose axes include prompt length, requested output length, arrival rate, concurrency, cache state, context allocation, sampling, and the latency objective. Changing one coordinate can move the bottleneck without changing the model or runtime.

A batch summarization worker values completed tokens over minutes. An interactive assistant values the first useful token and a steady stream. A classification endpoint may care almost entirely about short-prompt queueing. One scalar cannot rank these workloads because they do not optimize the same outcome.

This leads to the article's governing model:

Inference performance is the relationship between a declared workload, a resource configuration, and a service objective.

The benchmark must therefore describe that relationship rather than manufacture a universal speed score.

From arrival rate to saturation

At low load, a request can enter an idle server and begin work immediately. As arrivals increase, the scheduler combines or interleaves work, improving resource use. Eventually demand reaches the useful capacity of the configured slots and compute. New requests wait, tail TTFT rises, and timeouts or rejections may appear.

This creates a saturation curve. Before the knee, additional concurrency can improve total throughput without violating latency. Near the knee, small load increases produce disproportionate queueing. Beyond it, the server may emit more aggregate tokens while fewer requests satisfy the user-facing contract.

Utilization and latency do not rise in lockstep. Near capacity, ordinary variation in prompt and decode length has nowhere to go except a queue. The average can remain acceptable while a minority of requests absorbs most of the wait. Capacity planning therefore cares about the knee and its tail, not merely peak throughput.

Closed-loop tests can soften this signal because each slow completion delays the client's next request. Open-loop tests continue to offer work at a chosen rate and expose backlog growth more directly. The initial laptop experiment stays closed-loop for bounded complexity, but its conclusion is limited accordingly. It finds a concurrency envelope; it does not establish an arrival-rate capacity for production traffic.

A decision framework before a benchmark

Define four things before running a command:

1. Workload: prompt/output distributions, arrival pattern, and concurrency.
2. Contract: TTFT, TPOT or streaming-smoothness, error, and completion limits.
3. Controls: runtime, model, hardware, context, cache state, and sampling.
4. Decision: the configuration change the evidence will accept or reject.

If any item is missing, the benchmark can still produce numbers but cannot support an operational decision.

Prerequisites and implementation budget

The lab uses one local llama-server, one checksummed GGUF model, and a Python standard-library client. Allow 45–60 minutes after the model and runtime are available. Record the CPU model, physical and logical cores, memory, operating system, power mode, llama.cpp commit, build flags, model repository revision, artifact checksum, context size, thread count, and parallel slots.

The repository does not download a model, install a runtime, or send data outside localhost. The reader supplies and verifies both artifacts, keeping environment changes explicit.

Mental model: prefill, decode, and queueing

LLM inference has two visibly different phases. Prefill processes the input prompt and creates the key/value state used by generation. Decode produces output tokens autoregressively. Long prompts emphasize prefill; long answers emphasize decode. Combining both phases into one average hides which resource is constrained.

TTFT measures the user-visible delay from request submission to the first streamed output. It includes client, network, queueing, prompt processing, and initial decode work. TPOT approximates the interval between output tokens after the first one. Inter-token latency is the more detailed distribution; TPOT is a useful aggregate when the client cannot reliably map every stream event to one token.

This decomposition makes the metrics diagnostic. Rising TTFT with stable TPOT is consistent with more time in admission, queueing, or prefill. Rising TPOT points toward decode or scheduling interference. Neither proves a cause, but each narrows the server-side evidence to inspect.

Throughput answers “how much work completed?” Goodput answers “how much work completed inside the service contract?” If ten requests finish but only four meet the TTFT and TPOT limits, throughput counts ten while goodput counts four. That distinction prevents a saturated server from looking healthy merely because it continues emitting tokens.

Define the workload before measuring

A benchmark is a tuple, not a product ranking:

• server runtime and build;
• model and quantization;
• hardware and power state;
• prompt and output-length distributions;
• context, batching, thread, and parallel-slot settings;
• arrival pattern and concurrency;
• latency and error objectives.

The recorded experiment deliberately fixes one short prompt—“Explain why idempotency matters in two sentences.”—and a maximum output of 128 tokens. It submits eight requests at concurrency 1, 2, 4, and 8 after warmup. That narrow workload makes the saturation mechanism visible; it does not represent a production prompt-length distribution.

Closed-loop concurrency keeps a fixed number of requests active. It is useful for finding the server’s capacity envelope but can hide overload because a slow response delays the next submission. A fixed-rate test is a later extension: it keeps offering requests even when the server falls behind. This lab uses closed-loop load to stay within a laptop-sized experiment.

Experiment: test the saturation hypothesis

The scaffold uses urllib.request rather than an SDK. For every request it writes one JSON object containing a stable request ID, client-side executor wait, TTFT, TPOT, end-to-end latency, completion-token count, stream-event count, and any error. A summarizer calculates p50 and p95 latency plus the number of requests satisfying the declared TTFT and TPOT bounds.

git clone https://github.com/ihiteshsharma/llm-inference-real-hardware.git
cd llm-inference-real-hardware
python3 -m unittest -v test_lab.py

python3 bench.py run \
  --base-url http://127.0.0.1:8080 \
  --model qwen2.5-3b-instruct \
  --requests 8 \
  --concurrency 1 \
  --max-ttft-ms 1500 \
  --max-tpot-ms 120 \
  --output results/short-c1.jsonl

The thresholds above are lab inputs, not universal targets. They must be chosen before viewing results and justified by the intended interaction. A background extraction job may tolerate seconds of TTFT. An interactive coding assistant may not.

Client executor wait is not server queue time. The llama.cpp server exposes a Prometheus-compatible /metrics endpoint when started with --metrics, including processing and deferred-request gauges. Those metrics provide server-side evidence of queueing. The client record and server metrics should be reported separately instead of subtracting timestamps and naming the remainder “queue time.”

Verification and observed results

Before interpreting performance, verify accounting invariants:

• submitted requests equal completed plus failed requests;
• successful streamed requests have a first-output timestamp;
• TTFT, TPOT, and end-to-end durations are non-negative;
• the JSONL file can reproduce every percentile in the summary;
• model, server, context, workload, and hardware pins accompany the result.

The executed reference used an Apple M2 Pro with 16 GiB unified memory, four CPU threads, zero GPU layers, llama.cpp release b10217 at commit ddd4ec142, and the official Qwen2.5 3B Instruct Q4_K_M artifact. Its SHA-256 is 626b4a6678b86442240e33df819e00132d3ba7dddfe1cdc4fbb18e0a9615c62d.

The service objective was declared before inspection: every request had to complete, TTFT p95 had to remain at or below 1500 ms, TPOT p95 at or below 120 ms, and goodput had to equal the eight submitted requests.

Concurrency 2 was the highest observed passing point. Every request still completed at concurrency 4 and 8, so an availability-only dashboard would have called the server healthy. Goodput rejected those runs because six requests breached the TTFT contract.

TPOT p95 remained between 25.900 and 27.666 ms while TTFT p95 rose by more than 6.9 seconds from concurrency 1 to 8. In this fixed one-slot experiment, that separation is consistent with work waiting before generation rather than decode becoming progressively slower. It is a bounded observation, not proof that every TTFT regression is queueing. llama.cpp's processing and deferred-request metrics provide the server-side evidence needed for that diagnosis.

Failure injection and recovery

The negative test offered concurrency 4 against one configured server slot. All requests completed, but TTFT p95 reached 3066.273 ms and only two of eight requests counted as goodput. That is the operational failure: completed work no longer met the user-facing contract.

Recovery returned offered concurrency to 2 and reran the same prompt, output limit, model, and server configuration. All eight requests again passed both latency gates; recovery TTFT p95 was 991.231 ms and TPOT p95 was 23.988 ms. A successful recovery is not merely “the process stopped crashing.” It restores the declared goodput and error limits.

Observability and debugging

Preserve raw JSONL rather than only a chart. Record server startup logs, model-load time, process memory, prompt and predicted token counters, active and deferred requests, and termination reason. Use a monotonic clock for durations. Wall-clock timestamps remain useful for correlation but can jump.

When results vary, first check warmup separation, power throttling, competing applications, model or build drift, and prompt-length drift. Then inspect server queueing and prompt/decode metrics. Repeating an uncontrolled benchmark produces more samples of the same ambiguity.

Security and isolation

Bind the lab to 127.0.0.1. The local server has no production authentication, authorization, TLS termination, abuse controls, or prompt-log redaction. Do not use confidential prompts. Verify model provenance and checksum before loading the artifact, and review its license before reuse.

Remote exposure requires an authenticated proxy, request-size and concurrency limits, log redaction, network policy, and tenant isolation. OpenAI-compatible request syntax is not a security boundary.

Cost and scaling

On a local CPU, the immediate cost is reserved memory, CPU time, energy, and contention with other workloads. In production, compare cost per good request, not cost per generated token. A cheaper replica that misses the latency objective can be more expensive once retries, abandonment, and overprovisioning are included.

This laptop experiment does not justify claims about GPUs, multiple replicas, or autoscaling. It does establish the workload definition and measurement contract needed to compare them later.

What the benchmark does and does not claim

The experiment supports one conclusion: for the pinned host, model, prompt, and one-slot configuration, concurrency 2 was the highest tested SLO-compliant load, overload was visible in TTFT and goodput, and reducing offered load restored compliance.

It does not turn one host into a universal engine ranking. It does not compare differently quantized models, mix chat templates, or infer model quality from speed. It does not label client executor delay as server queue time, and it does not discard non-compliant requests from the coverage report. The Apple M2 Pro measurements describe that machine's behavior, not a general CPU result.

This boundary is useful. A narrow benchmark with raw records can be repeated on another runtime or deployment class. A broad “fastest server” conclusion built from uncontrolled inputs cannot.

Alternatives and trade-offs

llama-bench is the right tool for low-level prompt-processing and token-generation throughput under controlled parameters. The client harness is complementary: it measures an application-visible HTTP and streaming path. vLLM’s serving benchmark provides a richer ready-made workload driver and standardized metrics when vLLM is the target runtime. General HTTP load generators can generate concurrency, but most do not understand streamed tokens, TTFT, or token-aware goodput without custom instrumentation.

The minimal standard-library harness is intentionally narrow. Replace it when trace replay, tokenized input distributions, open-loop arrivals, or coordinated multi-host load becomes necessary.

Productionization gap

A production benchmark must replay representative traffic, preserve errors, separate cold and warm paths, and model network proxies, admission control, heterogeneous replicas, autoscaling, multi-tenancy, prompt-cache affinity, and tokenizer tiers. It must also track quality: a server that is fast because it truncates output or changes decoding behavior has not improved the service.

The local artifact proves measurement mechanics, not fleet capacity. Its useful output is a versioned workload, raw evidence, declared SLO, and recovery point that a larger environment can reuse.

Cleanup

Stop llama-server. Remove generated JSONL after preserving reviewed evidence. Delete downloaded GGUF artifacts when they are no longer required, and record what was removed. No cloud resource is created by this lab.

Sources

• llama.cpp server reference and metrics
• llama.cpp benchmark tool
• vLLM serving benchmark
• vLLM production metrics
• NVIDIA LLM inference metric definitions