Build a Lexical Search Baseline Before Adding Embeddings
Use lexical retrieval as the experimental control for judging when dense, hybrid, or reranked search actually improves evidence selection.
Embeddings, hybrid fusion, and reranking can make a retrieval architecture look sophisticated before anyone knows whether it retrieves better. Without a measured control, a new stack may lose exact error codes, product identifiers, or domain terms while still returning plausible documents. It may also hide ingestion defects because the result page is never empty.
A lexical baseline is not the primitive system that semantic search replaces. It is the experimental control that tells a team what the next retrieval layer actually improves.
This article builds that control from first principles, then uses a small SQLite FTS5 experiment to measure ranking, remove one judged-relevant document, detect the regression, rebuild the index, and preserve an intentional synonym failure for the next curriculum step.
Complete source code: ihiteshsharma/lexical-search-baseline-lab
Clone the verified implementation or use it to compare each step in the experiment.
Retrieval selects evidence
Retrieval is candidate selection. Given an information need and a corpus, it ranks a small set of documents or passages for a downstream consumer. In RAG, generation cannot recover evidence the retriever never supplied. A fluent answer therefore says little about retrieval quality: the model can respond plausibly to weak context.
Lexical retrieval compares query terms with indexed terms. It is especially strong when the terms carry precise identity:
- error codes such as
ERR_CONN_RESET; - product names and part numbers;
- API symbols and configuration keys;
- names, dates, and domain-specific phrases;
- rare terms whose presence is highly discriminative.
Dense retrieval represents queries and documents as vectors and compares their geometry. It can connect paraphrases that share meaning without sharing words. That strength creates different failure modes: exact identifiers may be diluted, embedding changes can alter rankings globally, and debugging a similarity score is less direct than inspecting matched terms.
The engineering choice is not lexical or semantic by ideology. It is which measured channel covers the workload's information needs.
The governing model is:
Retrieval quality is a property of a corpus, query distribution, judgment set, cutoff, and ranking system—not of a retriever name in isolation.
A baseline reveals whether a new channel recovers missing candidates, improves ordering, or merely changes results.
The term-document model
A lexical index records which terms occur in which documents. A query activates a subset of those terms; the ranking function estimates which documents best explain them.
Three intuitions drive BM25-style scoring:
Term frequency: repeated occurrence can indicate that a document is about the term, though the value should saturate rather than grow forever.
Document frequency: a rare term is more informative than a word appearing everywhere.
Length normalization: long documents contain more opportunities for accidental matches, so raw counts need adjustment.
SQLite FTS5 exposes a built-in bm25() function and maps its hidden rank column to BM25 by default. Its implementation returns better matches with numerically smaller values, an inversion documented by SQLite. Those raw magnitudes are local implementation details; they are not portable relevance scores or probabilities.
Analyzers decide what enters this model. Tokenization, case folding, stemming, diacritics, punctuation, and synonyms can change the effective corpus even when source text is unchanged. A lexical baseline therefore pins analyzer behavior as carefully as an embedding system pins its model.
Matching is not understanding and does not need to be
Lexical ranking does not construct a general representation of meaning. That limitation is often presented as a reason to skip it. In production search, the limitation is also a source of control.
Matched terms can be inspected. Analyzer changes can be versioned. Exact identifiers can be tested deterministically. A query that fails because it uses a different word is easier to classify than a vector result that moved after an opaque model update. This makes lexical retrieval a strong diagnostic channel even when it is not the final ranking system.
Semantic similarity has the opposite trade-off. It can connect “car” with “automobile” or a natural-language symptom with a differently worded runbook. It can also retrieve thematically related but operationally wrong content. Similarity does not imply relevance, authorization, freshness, or factual compatibility with the query.
The two approaches encode different evidence. Lexical scores say terms and their corpus statistics support this match. Dense scores say learned representations place the items nearby. Hybrid retrieval is useful when the judged workload needs both—not because combining two systems is inherently more advanced.
Source and index are different artifacts
The source table answers what should be searchable. The FTS structure is a derived representation used to retrieve it. Treating them as the same thing hides ingestion failures.
An external-content FTS5 table can fetch displayed content from a normal table while maintaining a separate index. Inserts, updates, deletes, or migrations must keep both consistent. SQLite documents a rebuild command that discards and reconstructs the full-text index from the content table.
This separation produces a useful operational invariant:
Every authoritative document expected by the retrieval contract must have the corresponding derived index state.
Counts alone do not prove correctness, but a source/index mismatch is a strong, cheap signal. Content hashes, tombstones, version fields, and reconciliation samples deepen that signal in production.
Judgments turn search into an experiment
A result list is not evidence of quality. Evaluation needs:
- representative queries;
- stable document identifiers;
- relevance judgments, commonly called qrels;
- a fixed cutoff such as the top three results;
- metrics and release thresholds chosen before a change.
Recall@k asks what fraction of judged-relevant documents appears in the first k results. It is valuable when missing evidence is the dominant risk. nDCG@k also considers graded relevance and position. It rewards placing highly relevant results above partially relevant ones.
Neither metric is sufficient alone. Perfect recall can coexist with poor ordering. Strong average nDCG can hide one catastrophic query. Always preserve per-query rankings, missed judged IDs, and deltas alongside the macro score.
Judgments are another model of reality. Weak assessors, ambiguous queries, tiny pools, leaked test cases, or query sets copied from implementation examples can produce confident but irrelevant metrics. A baseline is credible only when its queries represent the work the search system must perform.
Metrics can conceal the failure
Macro averages give every query equal weight. That is simple and often appropriate for a controlled lab, but production consequences may not be equal. Missing a documentation article and missing a safety procedure can have different costs. A weighted report or separate critical-query gate may be necessary.
Cutoff also encodes product behavior. Recall@3 matters only if the downstream system consumes three results. Measuring Recall@100 and then supplying five passages to a model inflates the apparent coverage. Likewise, nDCG rewards ordering within the chosen cutoff but cannot detect a relevant document that was never judged.
Judgment pools are incomplete. A new retriever may surface a genuinely relevant document absent from the qrels and be penalized as if it were wrong. Periodic pooling and assessor review are needed when candidate systems change substantially. Metrics should automate known expectations without freezing the corpus's understanding forever.
Finally, offline relevance says nothing about ingestion freshness, authorization, latency, or answer faithfulness. Those are separate gates. A higher nDCG system that serves stale or unauthorized passages is not a production improvement.
A decision framework for adding retrieval layers
Start with the smallest measured channel, then add complexity against held-out evidence:
- Lexical: control for identifiers, rare terms, transparent ranking, and low operational cost.
- Dense: add when judged paraphrase or semantic-match failures remain.
- Hybrid: combine channels when each recovers relevant documents the other misses.
- Reranking: add when candidate coverage is adequate but top-order quality is insufficient.
At every step, compare quality, latency, cost, ingestion reliability, authorization behavior, and debugging effort. An average metric gain does not justify a layer that violates access control or makes index freshness unobservable.
The baseline must remain runnable after the upgrade. Otherwise the team cannot tell whether future regressions come from the new layer or from a broken control.
For a dense comparison, preserve the corpus snapshot and stable IDs, then add embedding-model revision, chunking policy, normalization, distance function, and index parameters as new controls. For hybrid search, report each channel independently before fusion. A fused score without channel-level recall cannot show whether one retriever contributes evidence or merely rearranges results already found by the other.
Reranking should be evaluated only after candidate coverage is adequate. It cannot rank a document that no first-stage retriever supplied. This sequencing—coverage before ordering—keeps architecture aligned with the measured failure.
Experiment: build and break the control
The repository artifact is search_lab.py. It creates 20 synthetic documents, eight fixed queries, and graded qrels using Python and SQLite FTS5. The source lives in a normal table; the derived index uses an external-content FTS table; fts5vocab exposes indexed-document coverage.
git clone https://github.com/ihiteshsharma/lexical-search-baseline-lab.git
cd lexical-search-baseline-lab
python3 -m unittest -v test_lab.py
python3 search_lab.py
The hypothesis is that removing the only judged-relevant document for an exact identifier can leave the search API functioning while causing a measurable release-gate failure. Rebuilding from the authoritative source should restore the exact baseline.
Verification and observed behavior
Verification has three layers. Structural checks compare authoritative and indexed document coverage. Retrieval checks reproduce per-query rankings and aggregate metrics from fixed qrels. Recovery checks require rebuild to restore the exact baseline rather than merely produce a non-empty result page.
The executed run on CPython 3.14.6 and SQLite 3.53.2 returned verdict: passed: 20 source and 20 indexed documents across eight queries, macro Recall@3 of 1.0, and macro nDCG@3 of 0.958498. The exact ERR_CONN_RESET query retrieved DOC-007; the lease query ranked DOC-012 before DOC-013.
The script then deletes DOC-007 from the index without deleting it from the source. Unrelated searches continue to work, which is why this is a useful failure rather than a total outage.
That run left 19 indexed documents and observed Recall@3 fall to 0.875 and nDCG@3 to 0.833498. The gate failed. Rebuild restored 20 indexed documents, the original metrics, and the rankings. A synonym-only query for automobile returned nothing.
These observations are specific to the fixed synthetic corpus, qrels, ranking expression, and pinned runtime; they are not a general claim about lexical-search quality.
The supported conclusion is that this lexical control detects one silent index-loss regression and that deterministic rebuild restores the exact baseline. The unsupported conclusion is that embeddings are therefore superior: the empty synonym result identifies a representation limit, but no dense or hybrid candidate has been measured against the same judgments yet.
Interpret the failure correctly
The deleted identifier is an ingestion-integrity failure. Recovery is index reconciliation and rebuild.
The synonym miss is different. Source and index are consistent; the lexical representation simply lacks the relationship. Query expansion, a synonym map, dense retrieval, or hybrid fusion may address it. The correct next experiment judges the intended relevant document and compares another channel using the same query set.
If dense retrieval fixes the synonym case but loses error codes, hybrid search may be justified. If a reranker improves nDCG but not recall, it is reordering existing candidates rather than recovering missing evidence. The metrics identify which stage changed.
Observability, authorization, and deletion
Record source count, indexed count, corpus version, query-set version, per-query IDs, missed judgments, metric deltas, analyzer version, gate outcome, and rebuild duration. When quality drops, ask in order: did the source change, did derived index state change, or did ranking behavior change?
The synthetic lab has no access control. Production retrieval must apply document authorization before results reach a model or user. Filtering after top-k can leak metadata and can remove every useful result. Authorization behavior belongs in the offline evaluation set.
Indexes can retain deleted terms in internal structures. SQLite documents secure-delete behavior and compatibility trade-offs. Filesystem protection, database encryption, tenant isolation, deletion verification, and audit controls remain separate responsibilities.
Production implications and productionization gap
FTS5 is a deterministic local control, not a universal serving engine. Solr, Elasticsearch/OpenSearch, PostgreSQL full-text search, and managed services add analyzers, distributed indexing, filters, replicas, and operational tooling. Preserve corpus snapshots, queries, qrels, and metrics when changing engines.
Production evaluation also needs language coverage, spelling, duplicate handling, incremental ingestion, judgment pooling, assessor disagreement, query-log sampling, latency percentiles, load tests, shadow comparison, and rollback. Online clicks can supplement but not automatically replace offline judgments because position and presentation bias affect behavior.
The durable principle is that retrieval architecture should advance through measured failures. Embeddings are a candidate treatment, not the diagnosis.
A production review can use five questions:
- Which judged queries fail the lexical control, and why?
- Does another channel recover missing evidence or only change order?
- Are exact identifiers and authorization filters preserved?
- What latency and ingestion complexity buys the quality gain?
- Can the previous control be rerun during rollback?
If the answers are not visible, the new retrieval layer is an architectural preference rather than an evidence-backed improvement.
Cleanup
The lab creates a temporary database and removes it in finally. It uses no API key, paid service, or external corpus.