The index-type decision most teams get to skip, until they can't
Vector databases covers the database-level mechanics of storing and querying embeddings. This article is one layer up: the infrastructure and capacity-planning decisions that only surface once a vector index stops being "a table with a vector column" and starts being a distributed system in its own right -- which index algorithm to run, how to shard it across machines, and how to reason about the recall-latency-memory triangle every approximate index sits inside.
Most teams never need this layer -- a single-node index handles tens of millions of vectors comfortably. This article is for the point past that, where the shape of the data or the query volume forces real infrastructure decisions.
Index types and what they actually trade off
Flat (brute-force) search compares a query vector against every stored vector directly. It's exact -- 100% recall by construction -- and for a few hundred thousand vectors on modern hardware it's fast enough that there's no reason to reach for anything approximate. Past that scale, comparing against every vector for every query stops being affordable, and that's the point approximate methods exist to solve.
HNSW (hierarchical navigable small world) builds a multi-layer graph where each vector is linked to its approximate nearest neighbors; a query walks the graph from a coarse top layer down to a fine bottom layer, following the closest-looking edges at each step. It gives very good recall at very good query latency, at the cost of a larger memory footprint (the graph structure itself, on top of the vectors) and slower, more expensive inserts as the graph grows -- which is why the write path discussion in RAG pipeline design treats high-ingestion-volume systems as needing a separate staging strategy rather than writing straight into a large HNSW index.
IVF (inverted file index) clusters the vector space into a fixed number of partitions (via k-means or similar) at build time, and a query only searches the partitions closest to it rather than the whole index. It uses less memory than HNSW for the same corpus and inserts more cheaply, but recall is more sensitive to how well the partition count and search-time partition count (nprobe) are tuned to the actual data distribution, and a corpus whose distribution shifts after the index is built (partitions no longer reflect where the data actually clusters) needs periodic rebuilding to keep recall up.
Sharding an index across nodes
Past the point a single node can't hold the index in memory (or can't serve query volume fast enough alone), the index has to shard. Two shard strategies dominate, and they trade off differently.
| Strategy | How it works | Trade-off |
|---|---|---|
| Horizontal (by vector ID) | Each shard holds a disjoint subset of vectors; a query fans out to every shard and merges top-k results | Simple to scale (add a shard, redistribute), but every query touches every shard -- query cost grows with shard count |
| Semantic/clustered | Vectors are pre-clustered (e.g. by IVF-style partitioning) so semantically similar vectors land on the same shard; a query only fans out to the shards likely to contain a match | Lower per-query cost at scale, but rebalancing is expensive -- adding a shard means re-clustering, not just redistributing |
Horizontal sharding with full fan-out is the default because it's operationally simple and works correctly regardless of data distribution; it's usually the right starting point. Semantic sharding earns its complexity once fan-out cost (every shard doing work on every query, even ones with no relevant vectors) becomes the actual bottleneck -- typically well past a dozen shards.