When Q4 Passes Latency but Fails Quality

A smaller GGUF file is easy to deploy and easy to misunderstand. Lower precision can reduce weight memory and sometimes improve speed, but it can also damage task quality, behave differently across backends, and leave runtime or context memory as the real capacity constraint.

This tutorial compares Q4_K_M, Q5_K_M, Q6_K, and Q8_0 variants of one 3B instruction model while holding the runtime, hardware, workload, and sampling controls fixed.

The selector may return no winner. Deployment must distinguish “the only candidate measured” from “the smallest candidate satisfying the complete contract.”

Complete source code: ihiteshsharma/llm-inference-real-hardware
Clone the verified manifest, benchmark harness, and selector or use them to compare each step in the experiment.

Quantization changes representation

Model weights are numerical values learned during training. A higher-precision representation can express many distinct values; quantization maps them into a smaller set, usually with a scale and grouped encoding. The stored value becomes an approximation of the original. The difference is quantization error.

That error is not distributed uniformly through behavior. Tensors participate in different computations, and some perturbations matter more for a given input distribution. Mixed recipes such as K-quants use different treatment for selected tensors instead of assigning one identical encoding everywhere. An importance matrix adds calibration evidence about which weights or activations matter for representative text. It can reduce error for that distribution, but it cannot guarantee quality on unseen tasks.

Quantization error is task-shaped. Similar aggregate perplexity can conceal different behavior on structured output, ordering, arithmetic, tool selection, or domain language. A deployment consumes those particular behaviors, not an average model.

This is why “four bit” is not a complete technical description. It summarizes storage density while hiding grouping, scaling, tensor-specific choices, calibration, runtime kernels, and the behavior being preserved.

Compression and speed are different questions

Smaller weights reduce storage and often reduce memory traffic, which can help a bandwidth-bound decode loop. But runtime speed also depends on whether the backend has efficient kernels for that exact type, whether values require conversion, how tensors align, and which operations remain at higher precision. A smaller file can therefore be slower than a larger variant on a particular backend.

Memory has the same caveat. File size describes serialized weights. Resident memory also includes metadata, runtime buffers, and context/KV state. Quantizing weights may move a deployment from “does not load” to “loads,” yet leave concurrency limited by context memory.

Weight quantization does not automatically quantize activations, selected tensors, or KV-cache state. The format name describes an artifact recipe, not a live server's complete memory layout.

The governing model is a constrained optimization:

Minimize deployment cost or memory while satisfying predeclared task-quality and latency constraints on the target hardware.

Choose the rule before seeing results

The experiment needs an ordered decision, not an after-the-fact score:

1. Fix a quality floor using held-out tasks relevant to the deployment.
2. Fix TTFT, TPOT, and failure limits from the serving contract.
3. Reject every artifact that fails either class of constraint.
4. Among the survivors, select the smallest artifact or lowest measured cost.
5. If none survives, retain the previous control and report no qualifying candidate.

This rule prevents a visually attractive speed or memory result from negotiating away quality after measurement. It also produces a decision that can be replayed when the model, runtime, hardware, or workload changes.

Prerequisites and implementation budget

This is the third article in the Production LLM Inference on Real Hardware series and assumes Articles 1 and 2. The Q4-first stage takes 30–45 minutes; the four-variant comparison takes 60–90 minutes plus downloads. It uses one pinned llama.cpp build, the official Qwen GGUF repository, and the shared standard-library harness.

Four model downloads can consume several gigabytes. Confirm disk and memory budgets before starting. Record the model repository revision, exact filenames, sizes, SHA-256 checksums, llama.cpp commit and build flags, CPU and memory, context, parallel slots, thread count, prompt set, output target, and sampling configuration.

Mental model: the feasible region

Quantization reduces the precision used to represent weights. GGUF names such as Q4_K_M are recipes, not a complete prediction of behavior. K-quant recipes may use different tensor types for different parts of the model. Importance-matrix-informed quantization can preserve selected behavior by using calibration data to guide precision allocation.

Three budgets interact:

1. Memory: file size, resident weight memory, runtime buffers, and context state.
2. Performance: load time, prompt processing, TTFT, TPOT, and goodput.
3. Quality: whether outputs satisfy the intended tasks and regression checks.

Selecting only by file size optimizes one observable. Selecting only by perplexity optimizes a broad language-model measure that may not represent the deployment task. Selecting only by tokens per second may reward a model that is faster because it fails instructions.

Valid candidates form a feasible region satisfying quality, latency, compatibility, and memory constraints. Only inside that region should size or cost be optimized. A faster point outside the quality boundary is not a winner; it is invalid under the declared contract.

Control the comparison

The valid experiment changes one variable: the GGUF variant. Everything else stays fixed:

• same source model family and tokenizer;
• same official repository revision;
• same llama.cpp commit and backend;
• same CPU, threads, context, slots, prompts, and output limits;
• same temperature, seed where supported, and stopping behavior;
• same warmup and measured repetitions;
• same held-out cases and rubric.

The official Qwen model card publishes Q4_K_M, Q5_K_M, Q6_K, and Q8_0 artifacts for the same 3B instruction model. Their filenames still require checksums and a repository revision. A mutable “main” URL is not enough for a result intended to survive later updates.

Design the held-out quality set

The local evaluation should be small enough to inspect and broad enough to reject a clearly unsuitable quant. The six fixed cases use stable IDs and four bounded task types relevant to a small instruction model:

• deterministic JSON extraction;
• retry and durable-replay semantics;
• deterministic lexical ordering and arithmetic;
• diagnosis of an inference-latency symptom.

Write the rubric before running any model. Each case should have an input, expected constraints, and a deterministic pass function where possible. Avoid asking an LLM judge to decide everything; that would introduce another model and another source of variance. Human review can supplement the deterministic gate for ambiguous language quality.

Keep the cases held out from any importance-matrix calibration text. Using evaluation cases to guide quantization would contaminate the comparison.

Build the artifact manifest

quant-manifest.example.json lists the four variants and release thresholds. Before execution, copy it to quant-manifest.json and replace every pin marker:

{
  "repository_revision": "<resolved revision>",
  "llama_cpp_commit": "<commit>",
  "thresholds": {
    "min_quality_pass_rate": 0.8,
    "max_ttft_ms_p95": 1500,
    "max_tpot_ms_p95": 120
  }
}

These example thresholds are not production recommendations. They demonstrate that the rule is chosen before results. The final values must reflect the service contract established in Article 2.

For each artifact, collect file size, resident memory, load time, prompt throughput, generation throughput, TTFT, TPOT, goodput, and held-out pass rate. Preserve per-case outputs so a score can be audited.

Experiment: test the first candidate without forcing a winner

Start llama-server with the first variant and the exact Article 2 configuration. Run warmup separately, then execute the shared prompt matrix and held-out cases. Stop the server, record cleanup and memory recovery, and repeat for the next variant.

git clone https://github.com/ihiteshsharma/llm-inference-real-hardware.git
cd llm-inference-real-hardware
python3 -m unittest -v test_lab.py
cp quant-manifest.example.json quant-manifest.json

quant_compare.py joins observations to the manifest, evaluates the predeclared gates, and selects the smallest passing artifact by recorded bytes only after every manifest variant has an observation.

python3 quant_compare.py \
  --manifest quant-manifest.json \
  --observations results/observations.json

Verification and observed decision

An eventual cross-quant comparison is valid only when:

• all four artifacts resolve to the same source model family and repository revision;
• checksums, sizes, runtime, hardware, and controls are recorded;
• all variants expose the same API contract;
• each held-out case joins by stable ID and malformed or missing output fails;
• warmups are excluded from measured results;
• quality thresholds were fixed before inspection;
• raw results reproduce the selected variant.

The executed first stage produced this observation:

[
  {
    "name": "Q4_K_M",
    "quality_pass_rate": 0.666667,
    "ttft_ms_p95": 515.831,
    "tpot_ms_p95": 62.793
  }
]

The Q4_K_M file was 2,104,932,768 bytes with SHA-256 626b4a6678b86442240e33df819e00132d3ba7dddfe1cdc4fbb18e0a9615c62d. The expanded server used four 4096-token slots, four CPU threads, and zero GPU layers. At offered concurrency 4, TTFT p95 was 515.831 ms and TPOT p95 was 62.793 ms, both inside the declared 1500 ms and 120 ms limits.

Quality did not pass. Four of six cases met their written criteria, giving 0.666667 against a floor of 0.8. The model produced v1, v2, v10 instead of the requested lexical order v1, v10, v2. In the latency-diagnosis case it gave a vague pipeline answer without identifying queueing or prefill as the dominant cause. The JSON extraction, idempotency, arithmetic, and durable-replay cases passed.

The selector reported:

{
  "verdict": "incomplete",
  "coverage_complete": false,
  "observed_variants": ["Q4_K_M"],
  "missing_variants": ["Q5_K_M", "Q6_K", "Q8_0"],
  "selected": null
}

This is the correct result for two independent reasons. Q4_K_M failed the quality floor, and a one-variant run cannot compare the manifest. Lower precision often reduces artifact size, but quality and speed need not follow a monotonic curve across backends or tasks. Nothing in this observation predicts whether Q5_K_M, Q6_K, or Q8_0 will qualify.

Failure injection and recovery

The failure exercise is the executed Q4 result: the artifact fits, the expanded serving envelope meets its latency gates, and the release selector still rejects it because quality is below 0.8. Fit and speed are necessary gates, not permission to waive task correctness.

Recovery does not lower the threshold after observing the miss. It preserves the failed Q4 record, executes Q5_K_M next under the identical controls, and continues through Q6_K and Q8_0. Only complete coverage can select the smallest passing artifact. If none passes, recovery is to retain the prior approved model or change the deployment requirement—not manufacture a winner.

A second useful negative test changes one checksum in the manifest. The execution wrapper should refuse to load the mismatched artifact. This protects the experiment from silent file replacement; it is not a substitute for publisher and license review.

Observability and debugging

Capture model-load logs, GGUF metadata, resident memory, prompt/predicted token metrics, per-request latency, server deferral, process CPU, and per-case quality outcomes. When two runs disagree, check artifact hashes, build drift, warmup, sampling, prompt serialization, chat template, thermal state, and background load before attributing the difference to quantization.

Do not average away a systematic task failure. Preserve case-level regressions and identify whether they cluster around formatting, instruction following, factual recall, or long prompts.

Security and isolation

Bind every server to localhost. Verify repository ownership, revision, checksum, and model license. Model files are executable inputs to a complex parser/runtime boundary and should be handled as supply-chain artifacts. Do not include confidential prompts or production documents in the held-out set.

Production artifact promotion needs a reviewed registry, access control, malware and vulnerability processes, provenance metadata, and rollback to a previously approved checksum.

Cost, scaling, alternatives, and trade-offs

The smallest qualifying quant may reduce memory per replica, permit more replicas on a host, or fit hardware that higher precision cannot. Those are deployment benefits only when quality and goodput remain acceptable. Include download/storage, load time, memory headroom, CPU energy, and operator effort in the decision.

Perplexity with llama-perplexity can add a broad language-model signal. KL-divergence or importance-matrix experiments can deepen analysis. They are optional here because the tutorial’s primary gate is an auditable task set plus serving performance. AWQ and GPTQ address different runtime ecosystems and would turn this laptop experiment into an engine-format survey.

What this negative result supports

The experiment supports a narrow conclusion: on the recorded Apple M2 Pro workload, Q4_K_M satisfied the latency gates at offered concurrency 4 but did not satisfy the fixed six-case quality gate. It therefore is not approved by this decision rule.

It does not establish that Q4_K_M is generally poor, that another quant will pass, or that a larger file is always better. The cases are a small deployment-shaped regression set, not a general model benchmark. A small performance difference within run-to-run variation would also require repeated controlled runs before promotion.

Productionization gap

A small local held-out set cannot represent every domain, language, long-context behavior, safety requirement, or future prompt distribution. Production needs a versioned evaluation dataset, protected holdout, online shadow or canary comparison, tenant-specific quality checks, cost accounting, artifact promotion controls, and rollback.

Hardware-specific kernels can change the frontier. A decision on one CPU does not rank variants on another CPU or GPU. Repeat the controlled experiment on the actual deployment class before promotion.

Cleanup

Stop each server between variants and confirm its listener and memory are released. Retain manifest, hashes, raw measurements, case outputs, and the final decision record for review. Remove multi-gigabyte GGUF files when they are no longer needed and record which checksummed artifacts were deleted.

Sources

• Official Qwen2.5 3B Instruct GGUF model card
• Qwen quantization with llama.cpp
• llama.cpp quantize reference
• llama.cpp benchmark reference
• llama.cpp perplexity tooling