OpenSearch vector search looks like one more query clause: put a knn block on a knn_vector field and it returns the nearest documents by embedding distance. What sets it apart from its Lucene-based cousins is a deliberate engine choice — the k-NN plugin can back the same field with Lucene’s native HNSW, with nmslib, or with Faiss, each with a different math, memory profile, and feature set. That decision, plus the space type, the algorithm parameters, and how hard you quantize, sets your recall, your latency, and the size of cluster you pay for. This piece works through the numbers: the engines and their score conversions, HNSW versus IVF, exact fallback, filtering, Faiss quantization, and the native-memory model that decides the bill.
Three engines behind one knn_vector field
A knn_vector field with index.knn: true builds an approximate-nearest-neighbour structure that stands in for exact k-NN — scoring one query against all N vectors of dimension d is O(N * d), billions of multiply-adds and tens of gigabytes at scale, which approximation dodges by touching a tiny fraction of the points. The distinguishing fact about OpenSearch is that knn_vector is a front for three interchangeable backends, chosen per field under method.engine:
| Engine | Methods | Character |
|---|---|---|
lucene | HNSW | Pure-Java, no native memory, tight OpenSearch integration, filtering built in |
faiss | HNSW, IVF | Native library, richest quantization (SQ, PQ), disk mode, IVF for huge sets |
nmslib | HNSW | The original engine, now deprecated — migrate to Faiss or Lucene |
The split that matters: Lucene stores its graphs the way Lucene stores everything, so they ride the filesystem cache. Faiss and nmslib are native libraries whose graphs live in off-heap native memory that OpenSearch loads and budgets separately. Choosing an engine is choosing a memory model and a feature set, not just an implementation.
Space types and how a distance becomes a score
The space_type fixes the similarity metric, and OpenSearch always reports a bounded, larger-is-better score, so each raw distance is passed through a monotone conversion:
l2 : d = Σ(a_j - b_j)^2 score = 1 / (1 + d)
cosinesimil: c = (a·b)/(|a||b|) score = (1 + c) / 2
innerproduct: ip = a·b score = ip + 1 (ip ≥ 0)
score = 1 / (1 - ip) (ip < 0)
l1 / linf : Manhattan / Chebyshev score = 1 / (1 + d)Two consequences follow. Cosine similarity is just inner product on normalized vectors, so if you L2-normalize embeddings at index and query time you can use the cheaper innerproduct space and skip the per-comparison norm. And the conversion is monotone, so it never reorders results — it only maps distances into a comparable band you can threshold or blend with a BM25 score.
HNSW parameters: m, ef_construction, ef_search
All three engines can build Hierarchical Navigable Small World graphs — a layered proximity graph whose sparse upper layers are long-range sketches and whose layer 0 holds every node, giving a roughly log_m(N) greedy descent instead of an O(N) scan. Three method parameters tune it:
m (def 16) neighbours/node → permanent graph size + recall
ef_construction (build) insert-time beam → graph quality, one-off cost
ef_search (query) search-time beam → recall vs latency, live dialThe asymmetry is the tuning story: only ef_search is cheap to change after indexing, so it is the knob you turn on a slow query. m and ef_construction are baked in at build and need a reindex to change.
Faiss IVF: coarse quantization instead of a graph
Faiss adds a second method HNSW cannot match at very large N: an inverted file (ivf). Rather than a proximity graph, IVF runs k-means to learn nlist centroids, assigns every vector to its nearest centroid, and at query time probes only the nprobe closest lists:
train : k-means → nlist centroids (needs a training sample)
search: rank centroids by distance, scan the nprobe nearest lists
cost ≈ O(nlist * d) + O((nprobe / nlist) * N * d)IVF trades a mandatory training step and a recall knob (nprobe) for a far smaller memory footprint: it stores no per-node edge lists, only centroids plus the vectors. HNSW usually wins on recall-at-latency for moderate corpora; IVF wins when N reaches the hundreds of millions and the graph’s edge memory no longer fits in RAM.
Exact search with the knn_score script
Approximate search is the wrong tool when the candidate set is already tiny or when you need a provably correct top-k. For that OpenSearch exposes a brute-force path: a script_score query using the Painless knn_score function, which scores every document that passes the surrounding filter against the query vector at full precision.
script_score { filter q } → for each surviving doc: sim(query, doc.vector)
cost = O(M * d) where M = docs passing the filter, not NThis needs no index.knn or built graph at all — the vectors can be stored plain. It is exact and the right answer whenever a selective filter leaves only a few thousand candidates: scanning M vectors directly beats walking a graph built for all N. It is the wrong answer over millions of unfiltered documents, where its linear cost is exactly what HNSW exists to avoid.
Filtering: the pre-filter that traverses the graph
Real queries want nearest neighbours among documents that also match a predicate — a tenant id, a price band, a date range. Post-filtering (run k-NN, then drop non-matches) can return too few results. OpenSearch’s efficient filtering instead pushes the predicate into the traversal — the graph walk only accepts nodes in the matching bit set.
The important caveat is engine-specific: efficient filtering is supported by the Lucene and Faiss engines, not by deprecated nmslib. And a graph built over all N points has edges chosen for the unfiltered space, so a very selective filter fragments the reachable subgraph. OpenSearch handles that the way you would hope: when the filtered set is small relative to the approximate work, it falls back to an exact scan over just the matching documents — faster and exactly correct on a tiny survivor set.
Faiss quantization: fp16, scalar, and product
The vectors, not the graph, dominate memory, so Faiss’s encoders are the highest-leverage knob OpenSearch gives you. Each maps float32 components onto something smaller and stores the reduced form in the index:
fp16 (SQ) : clip to [-65504, 65504], store 16-bit half ~2x smaller
int8 (SQ) : q_j = round((v_j - lo) / (hi - lo) * 255) ~4x smaller
PQ : split d into m subvectors, k-means codebook up to ~64x smaller
each subvector → one byte code (nbits = 8)Scalar quantization (fp16, int8) is nearly lossless and needs no training. Product quantization is far more aggressive — it partitions each vector into subvectors and replaces each with its nearest codebook entry, so distances come from small lookup tables — but it requires training and trades measurable recall. The discipline is the usual one: compress until the working set fits in memory, then stop.
Disk-based mode: trading a rescore for RAM
When even int8 will not fit the corpus in memory, OpenSearch offers a disk-based mode (mode: on_disk) built on Faiss binary quantization: a heavily compressed index stays in memory for the graph walk while the full-precision vectors live on disk, read back only to rescore a handful of finalists:
compression_level 32x : ~1 bit/dim resident, fp32 on disk
1. HNSW walk on binary vectors → oversample candidates (in RAM)
2. rescore against fp32 vectors → one disk read per finalist
3. keep the true top kThe compression_level (up to 32x) sets how tiny the resident index is, and an oversample_factor controls how many candidates the cheap walk gathers before the disk rescore. It is the same ‘traverse cheap, rescore accurate’ pattern as scalar quantization, pushed to its limit so the footprint drops up to 32× while recall stays usable.
The native-memory model that decides your bill
Faiss and nmslib graphs live in off-heap native memory, and OpenSearch guards it with a circuit breaker (default 50% of the memory outside the JVM heap). Sizing reduces to estimating how much that graph plus its vectors will consume, and the plugin’s own formulas make it concrete:
HNSW : 1.1 * (4*d + 8*m) * N bytes
IVF : 1.1 * (4*d*N + 4*nlist*d) bytes
N = 20e6, d = 1024, m = 16:
fp32 HNSW : 1.1 * (4096 + 128) * 20e6 ≈ 93 GB native
int8 HNSW : ~1/4 of the vector term ≈ 25 GB nativeThe 1.1 is a ten-percent overhead allowance. Two moves follow: run the warmup API after a load so the first query does not pay the graph-loading cost, and quantize until the resident index sits under the circuit-breaker limit — once native memory is exhausted, graphs are evicted and reloaded per query and tail latency collapses.
Choosing an engine, and the pitfalls each invites
The engine decision drives everything else, so make it first.
| Want… | Pick |
|---|---|
| No native-memory tuning, simple ops, good filtering | Lucene HNSW |
| Maximum recall control, quantization, disk mode, huge N | Faiss (HNSW or IVF) |
| An existing pre-2.x index | nmslib — but plan a migration, it is deprecated |
The recurring failure modes all trace back to the math above. Picking nmslib loses efficient filtering. Using IVF without a representative training sample gives skewed centroids and quietly poor recall. Forgetting to normalize before an innerproduct space silently ranks by magnitude, not direction. Approximate search degrades silently — no error, just worse results — so keep a few hundred labelled queries and track recall against an exact knn_score baseline as a first-class metric.
knn clause, and its defining choice is the engine: Lucene keeps graphs in the JVM-managed filesystem cache with no native-memory tuning, while Faiss (and the deprecated nmslib) live in off-heap native memory but unlock IVF, richer quantization, and disk mode. The space_type maps every distance to a bounded larger-is-better score without changing rank — so normalize and use innerproduct to get cosine cheaply. ef_search is the one recall knob you can turn after indexing; m and ef_construction are baked in at build. Efficient filtering pushes the predicate into the graph walk (Lucene and Faiss only) and falls back to exact search on a small survivor set, the same job the knn_score script does directly. Faiss quantization — fp16 near 2×, int8 near 4×, PQ and disk mode up to 32× with rescoring — is the biggest lever on the native-memory bill: compress until the working set fits under the circuit breaker, warm the graphs, then measure recall against an exact baseline because it fails quietly.