Search System Architecture in Depth

Why architecture matters here

Search architecture matters because the quality-cost trade-offs are subtle. Bigger indexes, more sophisticated retrieval, ML reranking all cost latency and money. Product companies must balance freshness, relevance, cost, and latency.

Cost matters. Elasticsearch clusters at scale run into six-figure monthly bills; vector indexes add more.

Reliability is where analytics + retraining feedback loops shine. Click data trains reranker; refined queries improve future retrieval.

The architecture: every stage explained

Walk the diagram top to bottom.

Docs. Content to index — product data, articles, customer records.

Indexer. Tokenizes, normalizes (lowercase, stem), extracts features.

Inverted Index. Term → list of doc IDs + positions. The foundation.

Query. Parsed, expanded (synonyms), spell-corrected.

Retrieval. BM25 for keyword match; vector search for semantic; hybrid combines. Returns top-k candidates.

Reranking. ML model (learn-to-rank, LLM) reorders top candidates using richer features (query-doc similarity, freshness, popularity).

Personalization. User context (history, preferences) further reorders.

Result page. Snippets, highlights, facets, pagination. UI concerns.

Analytics. Click-through, dwell time, refinements. Feeds reranker training + query understanding.

Sharding + replication. Shard by doc ID or category; replicate for read scale + HA.

End-to-end query flow

Trace a query. User: "red running shoes for men size 10".

Query parser: extracts entities (color=red, category=running_shoes, gender=men, size=10). Expands "shoes" → "footwear."

Retrieval: BM25 against product name + description with entity filters. Simultaneously vector search on query embedding vs product embeddings. Fusion combines top 100.

Reranker (learn-to-rank model) scores each candidate with 40 features: keyword relevance, popularity, price, ratings, in-stock. Returns top 20.

Personalization: user's history shows preference for Nike; boost Nike results. Top 20 reranked.

Result page: snippets with query terms highlighted; facets for brand, price range, ratings.

User clicks 3rd result; spends 2 minutes; adds to cart. Analytics records: query, results shown, clicks, conversions. Feeds reranker retraining.

Sharded across 20 nodes by product ID; queries fan-out; results merged.

Inside the inverted index: term -> posting list -> intersection

An inverted index maps each term to a posting list: the ascending doc IDs that contain the term, usually carried alongside a per-document term frequency and the token positions needed for phrase queries. The term dictionary itself is kept sorted rather than hashed (Lucene stores it as a finite state transducer), which is why the same structure that answers title:shoes can also answer a prefix or range query without a second index.

Posting lists are delta-encoded and block-compressed. Instead of storing [17, 940, 941, 3002] you store the gaps [17, 923, 1, 2061] and bit-pack each block of 128 with the smallest width that fits. Common terms have small gaps, so they compress best per document -- the opposite of the usual intuition that frequent terms are the expensive ones.

A conjunctive query is an intersection. The engine orders the posting lists by length, iterates the shortest one, and for every candidate doc ID performs a skip-forward seek into the longer lists using the skip pointers stored between blocks. The cost tracks the shortest list, not the corpus size, which is why adding a highly selective filter term makes a slow query fast. A disjunction is instead a k-way merge driven by a priority queue over list heads.

Top-k retrieval does not score every match. Block-max WAND keeps the current k-th best score as a threshold and reads the maximum possible impact score stored per posting block; any block that cannot beat the threshold is skipped without being decompressed. On a head query this discards most of the list. Postgres implements the same shape internally -- see GIN inverted index architecture for its pending-list and fastupdate behaviour.

The analysis chain, and why index time and query time must agree

Analysis converts a raw string into the tokens that are actually stored. The chain has three stages: character filters (strip HTML, normalise punctuation), a tokenizer (split on Unicode word boundaries, or emit n-grams, or emit the whole string untouched for a keyword field), then an ordered list of token filters -- lowercase, ASCII folding, stopword removal, synonym expansion, stemming.

The most common relevance bug in production is an analyzer mismatch. Matching is exact string equality between the output of the index-time chain and the output of the query-time chain. If documents were indexed as run after stemming but the query chain emits running, the engine returns zero hits and reports no error at all. This also means changing an analyzer does not retroactively apply: the mapping for an analyzed field is immutable, so a stemmer change requires reindexing into a fresh index and swapping an alias.

Deliberate asymmetry is the exception, and it is how prefix autocomplete works. Index with an edge_ngram filter so "shoes" is stored as s, sh, sho, shoe, shoes; query with a plain analyzer so the user's "sho" is looked up as a single token instead of being expanded into its own prefixes. Express that with an explicit search_analyzer rather than hoping one chain does both jobs.

Because no single chain serves every purpose, model important text as a multi-field: an analyzed subfield for scoring, a keyword subfield for exact filters, sorting and facets, and often a third with a language-specific stemmer. Multi-fields cost index size, not query time.

Sharding, replication, and the scatter-gather query path

An index is split into shards, each a self-contained inverted index, and a document is routed to one of them by hash(routing_key) % primary_shard_count. That modulus is why the primary shard count is fixed at creation: changing it changes where every document belongs. The general partitioning tradeoffs are covered in database sharding architecture; what search adds is that the query, not just the write, must fan out.

A search with no routing key touches every shard. The coordinating node broadcasts the query, each shard runs the full retrieval and returns its own top-k as (score, doc ID) pairs, the coordinator merges those into a global top-k, and only then issues a second round trip to fetch the stored fields and build snippets for the surviving documents. This query-then-fetch split exists so the expensive part -- reading and highlighting document bodies -- happens for k documents rather than for shard_count times k.

Two consequences follow. First, latency is the slowest shard, not the average: with 20 shards you are sampling the p99 of your node fleet on every query, so garbage collection pauses and cold page cache on a single node set user-visible p99. Fewer, larger shards usually beat many small ones for this reason. Second, replicas add read throughput and availability but do nothing for a single query's latency, since a request still visits one copy of each shard.

Deep pagination: what scatter-gather does to page 500

Offset pagination and scatter-gather compose badly. To return results 10000 to 10020 with a global sort, the coordinator cannot ask each shard for 20 documents -- all 20 could live on one shard. Every shard must return its own top from + size, so page 500 across 20 shards means 20 x 10020 = 200,400 hits materialised and merged to hand back twenty rows. Cost grows linearly with page depth while the value of that page collapses.

Engines cap this rather than let it take the cluster down: Elasticsearch's index.max_result_window defaults to 10,000 and rejects deeper requests outright. The fix is a cursor rather than an offset. Sort on a key that includes a unique tiebreak, then pass the last row's sort values as the starting point:

{
  "size": 20,
  "sort": [ { "_score": "desc" }, { "doc_id": "asc" } ],
  "search_after": [ 8.42, "sku-99381" ]
}

Each shard now seeks directly past that point and returns 20 rows regardless of depth. The tiebreak field is mandatory: without a unique final sort key, ties straddle the cursor boundary and rows are silently duplicated or skipped. For full exports use a point-in-time reader or scroll, which pins a consistent segment view so concurrent indexing does not shift rows underneath the cursor -- at the cost of holding those segments from being merged away.

Relevance scoring: TF-IDF, BM25, and term frequency saturation

Classic TF-IDF scores a match as term frequency times inverse document frequency: a term is worth more when it appears often in this document and rarely in the corpus. Both halves are right in direction and wrong in shape. Raw term frequency is linear, so a page repeating "shoes" fifty times outranks a page that is genuinely about one pair of shoes, and long documents win simply by containing more words.

BM25 fixes both with two knobs. Term frequency enters as tf / (tf + k1) rather than tf, a saturating curve: with the usual k1 = 1.2, going from one occurrence to two buys a large score increase, and going from twenty to fifty buys almost nothing. That saturation is what makes BM25 robust to keyword stuffing and, more importantly, what makes multi-term queries behave -- a document matching both query terms once beats a document matching one term ten times, which is almost always the right answer.

The second knob, b (default 0.75), controls how strongly the score is normalised by document length against the field average. Set b = 0 and long documents dominate; set b = 1 and you fully penalise length, which is wrong for fields like a product description where length is not padding.

IDF is where sharding leaks into relevance: each shard computes document frequency from its own segments, so the same document can score differently depending on which shard it landed on. With millions of documents the distributions converge and nobody notices; with a small index, a filtered subset, or heavily skewed routing, scores become visibly inconsistent and you need a cluster-wide DF collection pass to stabilise them.

Two-phase retrieval: cheap recall first, expensive precision second

The economics are unambiguous. A cross-encoder that jointly encodes query and document is far more accurate than any lexical or bi-encoder score, and it costs one neural forward pass per candidate. At a few milliseconds each, ranking a hundred million documents per query is not slow, it is arithmetically impossible. So retrieval runs a cheap, recall-oriented first stage over the whole corpus and a costly, precision-oriented second stage over a few hundred survivors. The mechanics of the cross-encoder and the ranking losses that train it are worked through in reranking math.

The architectural point is the recall ceiling. A reranker can only reorder what it was given; a relevant document the first stage never returned is lost, permanently and invisibly. That is why the two stages are tuned against different metrics -- recall@k for retrieval, NDCG@10 for the reranker -- and why the honest way to debug bad results is to check whether the correct document was in the candidate set at all before touching the ranking model.

Reranking runs on the coordinator after the shard merge, not per shard. Running it per shard would multiply the model cost by shard count and still produce a locally optimal ordering. Coordinator placement also unlocks features that are unaffordable during retrieval: at k = 200 you can join click-through history, inventory, price, freshness and per-user affinity per candidate, which you could never do while streaming millions of postings. Budget accordingly -- a 300 ms envelope typically splits into roughly 40 ms retrieval, 30 ms merge and fetch, and the rest for the model, with k chosen to fill that remainder.

Vector search and hybrid retrieval with rank fusion

Lexical retrieval fails on vocabulary mismatch: "laptop won't charge" does not match a document about "notebook battery not powering on". Dense retrieval embeds query and document into a shared space and retrieves by nearest neighbour, which handles paraphrase but is weak exactly where lexical search is strong -- exact identifiers, part numbers, rare proper nouns, negation. Neither is a replacement for the other, so production systems run both.

Approximate nearest neighbour indexes are their own subject and are covered elsewhere in this site rather than re-derived here: HNSW for graph-based search, IVF-PQ for quantized inverted files, DiskANN for billion-scale graphs on SSD, and vector DB math for recall-versus-memory tradeoffs and pre- versus post-filtering. Vector database architecture covers the storage side.

What is specific to hybrid search is fusion. Naively adding a BM25 score to a cosine similarity does not work: BM25 is unbounded, corpus-dependent and not comparable between queries, while cosine sits in a narrow band. Any fixed weight you tune on one query set drifts on the next. Reciprocal rank fusion sidesteps calibration entirely by discarding the scores and combining ranks -- each list contributes 1 / (k + rank), conventionally with k = 60, and the contributions are summed per document. A document ranked first by one retriever and thirtieth by the other still outranks one ranked tenth by both, which is the behaviour you want when the two retrievers fail in different ways. RRF also needs no training data, which makes it the right default before you have enough labels for a learned fusion layer.

The indexing pipeline: segments, refresh, merges, and tombstones

A search index is not updated in place. Incoming documents accumulate in an in-memory buffer; on refresh that buffer is written as a new immutable segment and becomes visible to search. Elasticsearch refreshes every second by default, which is where "near real time" comes from -- a write is durable before it is searchable, and the gap is the refresh interval, not the commit. Durability is handled separately by a write-ahead translog that is fsynced on its own schedule, so a crash between refreshes loses nothing.

Because segments are immutable, a delete cannot remove anything. It sets a bit in a per-segment deleted-documents bitset -- a tombstone -- and the document keeps occupying its postings until the segment is rewritten. An update is a delete plus a fresh insert, so a hot document that is updated a thousand times leaves a thousand dead copies. Queries pay for this: deleted docs are still traversed during posting-list iteration and merely filtered from results, and stale entries continue to inflate document frequencies until they are reclaimed.

Reclamation is the job of the merge policy, which continuously combines smaller segments into larger ones and drops tombstoned documents in the process. Merging is the dominant source of background write amplification -- every document is rewritten several times over its life -- so a cluster that looks CPU-bound during ingestion is often merge-bound on disk.

Bulk and incremental loads therefore want opposite settings. For a backfill, disable refresh and replicas, load, then restore them so replication copies finished segments instead of duplicating the indexing work:

PUT /products/_settings
{ "index": { "refresh_interval": "-1", "number_of_replicas": 0 } }

// ... bulk load, batches of 5-15 MB per request ...

PUT /products/_settings
{ "index": { "refresh_interval": "1s", "number_of_replicas": 1 } }
POST /products/_forcemerge?max_num_segments=1

Force-merging to one segment is safe for an index that has stopped changing and actively harmful for one still taking writes, since the resulting huge segment is never merged again and its tombstones are never reclaimed.

Query understanding: spelling, relaxation, and the null-result path

The analysis chain owns stemming and synonyms because both must be symmetric with indexing. Query understanding owns the work that has no index-side counterpart, and it starts with spelling. Correction is not plain edit distance over a dictionary: candidates within edit distance two of a misspelling are numerous, and the right one is chosen by weighting corpus term frequency against what users actually searched. Query logs beat dictionaries here, because they contain the domain's proper nouns and brand names that no dictionary has. Correct silently for confident cases and offer "showing results for X, search instead for Y" when the margin is thin -- a wrong silent correction on a rare-but-real query is a worse failure than a missed one.

Relaxation is the other half. Long conjunctive queries fall off a cliff: "red running shoes for men size 10" as an AND over six terms matches nothing. Engines handle this with a minimum-should-match threshold that requires, say, 75% of terms rather than all, dropping the least informative (highest document frequency) ones first. The complementary risk is over-relaxation, where an already-good result set is diluted by partial matches -- so relaxation should trigger on low hit counts, not unconditionally.

Treat the null-result rate as a first-class metric with an owner. Every zero-result query is a user who leaves, and the distribution of them is the cheapest available roadmap: it names the synonyms, the products, and the intents your index does not know about.

Caching layers and what each one actually buys

Query traffic is Zipfian -- a small head of queries covers a large share of volume -- so a result cache keyed on the fully normalised query (analyzed terms, filters, sort, pagination cursor, and any personalization inputs) gets a high hit rate cheaply. The trap is the key: forget to include a filter or a user segment and you serve one user's results to another. Personalization and result caching are in direct tension, which is a good reason to cache the pre-personalization candidate list and apply user reordering after the cache.

Below that, the filter cache stores the bitset for a filter clause. Filters are binary and score-free, so status:active or a category constraint can be cached as a bitset per segment and reused across every query that mentions it, then intersected cheaply. This is the concrete payoff for expressing non-scoring constraints as filters rather than as query clauses.

Shard-level request caching is narrower than it looks: it is invalidated on every refresh, so on a one-second refresh interval it helps only indices that are not actively being written. It is genuinely valuable for aggregations over rolled-over time indices where older shards are frozen.

The cache that matters most is the one you do not configure. Posting lists and doc values are memory-mapped and served from the OS page cache; when the working set stops fitting in RAM, queries start hitting disk and p99 degrades by an order of magnitude with no code change and no error. Leave roughly half of node memory unallocated by the JVM for this, and treat the index-size-to-RAM ratio as a capacity metric rather than a storage one.

Evaluating search honestly: offline metrics versus online clicks

Offline evaluation runs a fixed query set against graded relevance judgments and reports NDCG@10 or MRR -- the formulas and a worked NDCG example are in reranking math. Its value is that it is fast, repeatable, and lets you compare two rankers before shipping either. Its limits are structural: judgments are expensive, they go stale as the corpus changes, the query set is almost never a representative sample of the real tail, and judges without the user's intent guess at ambiguous queries.

Online metrics have the opposite profile: real intent, real scale, but heavily biased. Position bias dominates -- the top result collects clicks largely because it is on top, so raw CTR partly measures your existing ranking rather than relevance. Presentation bias adds to it: a result with a thumbnail and rich snippet wins clicks against a better result rendered as plain text. Worst is selection bias: you observe clicks only on documents you chose to show, so training the next ranker on those logs teaches it to reproduce the current one, and genuinely better documents that today's ranker buries never get the impressions that would prove them.

The practical response is layered. Use interleaving rather than a split A/B for ranker comparison -- mixing two rankings within one result page and attributing clicks to their source controls for position and needs far less traffic than an A/B test. Instrument the whole session, not the click: click-then-immediately-return is a failure, and long dwell, add-to-cart or task completion are the signals worth optimising. Then watch the metrics no ranker can flatter -- null-result rate, query reformulation rate, and the share of sessions that end in a click below position three. Reserve a small randomised traffic slice to collect unbiased exploration data, and accept that a ranker trained purely on its own logs will always look better offline than it performs.