Qdrant Math + Architecture

Collections, shards, segments, replicas

A Qdrant collection is cut into shards (routed by a hash of the point id, or by an explicit shard key), and each shard into segments. A segment is the real index: its own vector storage, its own HNSW graph, its own payload store and payload indexes. New points land in an appendable segment that has no graph and is searched by brute force — which is why a freshly upserted point is queryable immediately, with no index lag. A background optimizer later seals segments, builds their graphs, and merges small ones.

Search fans out: a query with limit k runs independently against every segment of every shard, each returns a local top-k, and a coordinator merges. If each segment explores ef candidates, total distance work scales as S · ef. Sharding caps per-node memory near N/S vectors and runs graph walks concurrently, so p50 drops — but it adds no reduction in distance computations, plus a merge and a network hop. Shard for RAM and availability, not for CPU. A replication factor r stores S · r copies; with w write acks and read consistency c, the usual quorum rule w + c > r forces the read and write sets to overlap (r=3, w=2, c=2 is safe and survives one node down; w=c=1 is fast and may read stale).

Cosine, implemented as normalize-then-dot

Qdrant offers Dot, Cosine, Euclid and Manhattan. Cosine is the one with a trick behind it, because Qdrant never actually computes the formula at query time:

cos(x, y) = (x · y) / (||x|| · ||y||) = (x/||x||) · (y/||y||)

If every vector is L2-normalized once, at upsert, cosine collapses into a plain dot product. That is exactly what a Cosine collection does: it stores x/||x||, and every later comparison is a single SIMD dot product with scores bounded in [-1, 1]. Two things follow: the stored vector is not the one you sent, and magnitude information is gone for good.

The three knobs and the memory bill they imply

m (default 16) is edges per node per layer; layer 0 gets 2m. ef_construct (default 100) is the candidate list held while inserting a point — paid once, banked in the graph’s quality, free per query. hnsw_ef is the search-time beam width, set per request, and is roughly proportional to latency. m is the knob people under-set to save memory, and it is also the one that decides whether filtered search works at all.

HNSW draws each point’s level from a geometric distribution with P(level ≥ l) = m^(-l), so links per node are a geometric sum:

links/node = 2m + m · Σ_{l≥1} m^(-l) = 2m + m/(m − 1)
m = 16  →  33 links × 4 B  ≈  132 B per point

The hierarchy is nearly free: upper layers add about 3% to the link count, not 50% — layer 0 is the index. And graph cost is a fixed ~8m bytes per point, independent of dimension. For 5M vectors at d = 768:

raw float32 : 5e6 × 768 × 4 B = 15.4 GB      HNSW graph : 0.7 GB
int8 scalar :                     3.8 GB      binary     : 0.5 GB

The vectors are the bill; the graph is a rounding error. Compression is the only lever with an order of magnitude in it — and the graph does not shrink, so at binary precision it becomes over half your memory.

Why a filter breaks a navigable graph

This is the problem Qdrant is built around. Apply a filter — country = “DE” — and you are searching an induced subgraph: non-matching nodes are invisible. Delete enough nodes from any graph and it shatters into islands, and a greedy walk trapped in the wrong island quietly returns wrong answers. Percolation theory gives the threshold: keeping each node with probability p, a giant connected component survives while

p > p_c ≈ 1 / (⟨k⟩ − 1)   with  ⟨k⟩ ≈ 2m = 32  →  p_c ≈ 3%

That is the optimistic bound — random deletion from a random graph. Real filters correlate with position in embedding space, and HNSW’s graph is spatially structured rather than random, so the practical cliff arrives well above 3%. Halving m doubles p_c.

The cardinality estimator and full_scan_threshold

Because the right strategy depends on selectivity, Qdrant first estimates cardinality from payload-index statistics, then picks a plan:

The cut-over is full_scan_threshold, and its unit is the giveaway: kilobytes, not points. A linear scan is bandwidth-bound, so bytes touched is the honest cost. The 10000 KB default is ~3,300 points at d = 768 float32 but ~20,000 at d = 128 — against a graph walk that touches a few thousand vectors regardless of N. Quantize and the same byte budget covers 4× or 32× more points, quietly widening the band of filters that resolve as exact scans.

Filterable HNSW: payload_m and the tenant trick

For the awkward middle band Qdrant modifies the index itself. Where a payload index has values covering enough points, it builds additional HNSW links restricted to each subgroup, with degree payload_m. The subgraph for country = “DE” is then connected by construction and the walk cannot get marooned. You pay in build time and roughly 8 · payload_m extra bytes per point per grouping.

Multi-tenancy gets a sharper tool. Marking a payload index as a tenant key lets Qdrant physically co-locate each tenant’s points, so a tenant-scoped query reads one contiguous run of vectors instead of chasing random offsets across a 16 GB mmap. For SaaS workloads that single flag is usually worth more than any amount of hnsw_ef tuning.

Scalar quantization: the int8 arithmetic

Scalar quantization maps float32 to int8 per dimension, with bounds taken from a quantile of the observed distribution (default 0.99) so a few outliers cannot stretch the range and destroy resolution for everyone else:

δ = (hi − lo)/255      q_i = clamp(round((x_i − lo)/δ), 0, 255)
x̂_i = lo + δ · q_i      max error δ/2

The speedup beats the 4× memory saving, because the dot product factorises into δδ′ Σ_i q_i u_i plus three terms that each depend on only one vector. Only Σ q_i u_i is query-dependent, and that is an integer dot product a CPU executes 32 lanes at a time; the rest are per-vector scalars precomputed at index time. Four times less memory traffic plus integer SIMD is typically 2–4× faster, at a few points of recall.

Binary quantization: sign bits and popcount

Binary quantization keeps one sign bit per dimension: a 768-dim vector becomes 96 bytes, 32× smaller, and similarity becomes popcount(a XOR b) — twelve machine words instead of 768 multiplies. On a CPU-only box that is the largest single throughput lever available.

The justification is the random-hyperplane bound: for a roughly isotropic, zero-centred embedding, H/d ≈ θ/π, so cosθ ≈ cos(π H/d). The catch is that the hyperplanes here are the coordinate axes rather than random ones, so the bound holds only when no small set of dimensions dominates. Empirically, binary quantization is excellent at d ≥ 1024 for large well-trained encoders and unusable at d = 384.

Oversampling and rescoring

Quantized search is a filter, not an answer. The repair is two-stage: retrieve ⌈L · o⌉ candidates using the compressed vectors, rescore only those against the full-precision originals, keep the true top L. At L = 10 and oversampling o = 3 that is 30 × 768 ≈ 23k multiplies — nothing against a graph walk. It is what makes the aggressive modes usable: binary might recall 70% at o = 1 and climb into the 90s at o = 3–4. The real cost is I/O, one random read per candidate — so keep quantized vectors resident in RAM and originals on disk, or you have paid for compression without collecting the saving.

Pitfalls that follow from the math

Most Qdrant problems are one of these numbers being wrong. No payload index on a filtered field: the estimator is blind, the planner walks the graph with a rejection predicate, and recall collapses on exactly the selective filters you cared about. Lowering m to save memory: you save ~130 B per point and raise p_c, so filtered search degrades long before unfiltered search does. Binary quantization on a 384-dim encoder: isotropy fails and no oversampling rescues it — use int8. Too many segments: query work is S · ef, so heavy fragmentation quietly runs hundreds of searches per request.