Milvus is what a vector database looks like when it is designed for billions of embeddings and a cluster rather than a laptop. Where an embedded store runs in your process, Milvus takes the opposite bet: it is a disaggregated, cloud-native system that splits reading, writing, indexing, and storage into independently scalable pieces glued together by a log. The consequences ripple through everything — how a vector travels from an insert to a searchable index, why fresh data is queryable in milliseconds yet compact on disk minutes later, and how you dial the trade between freshness and latency. Underneath sit the same ANN engines everyone uses — IVF, HNSW, DiskANN — but Milvus wraps them in a segment lifecycle and a timestamp discipline worth understanding from first principles. This piece works through that machinery.
The disaggregated, log-first architecture
The defining choice in Milvus is that compute and storage are pulled apart, and every component is stateless where it can be. Instead of one process that owns the data and answers queries, Milvus is four layers. An access layer of stateless proxies terminates client requests and load-balances them. A coordinator tier assigns work, hands out timestamps, and tracks metadata. A tier of worker nodes — query, data, and index nodes — does the actual computing. And a storage tier holds the durable truth: etcd for metadata, a log broker (Pulsar or Kafka) for the write stream, and object storage (S3 or MinIO) for vectors and built indexes.
The glue is a log. A write is not applied in place; it is appended to the broker as a record, and the components that care subscribe to that stream. This ‘log as the system of record’ design means the durable state lives in shared storage, so a query node or data node can crash and be replaced without losing data — it just replays the log and reloads segments from object storage. Scaling reads means adding query nodes; scaling writes means adding data nodes; neither touches the other.
Collections, partitions, segments
Data in Milvus nests in three levels. A collection is the table-like top unit with a fixed schema and one vector field’s index and metric. A collection can be split into partitions — logical slices (by tenant, by date) that let a search skip whole regions of data by naming the partition, a cheap coarse filter. But the unit that actually matters for performance is the segment: collections and partitions are carved into segments, and a segment is what gets indexed, loaded, searched, and moved around.
A segment is a self-contained batch of rows — on the order of a few hundred megabytes to a gigabyte of vectors plus their scalar fields. Milvus builds one ANN index per sealed segment, loads segments into query-node memory to serve search, and distributes them across nodes for parallelism. A search over a collection is really a fan-out over its segments followed by a merge of the per-segment top-k results. Almost every operational behaviour — freshness, memory use, compaction — is a story about segments.
Growing segments vs sealed segments
Segments come in two states, and the distinction is the heart of how Milvus stays both fresh and fast. A growing segment is the mutable landing zone for new inserts. It lives in memory on query nodes as rows stream in from the log, and it has no ANN index yet — searching it means a brute-force scan of its vectors. That is fine because a growing segment is deliberately kept small, so the exact scan is cheap, and it is the reason a just-inserted vector is searchable almost immediately.
Once a growing segment reaches a size or time threshold it is sealed: declared immutable, no more rows accepted. Sealing unlocks the expensive optimisations that only make sense on fixed data — an index node builds a real ANN index over it, and the data is flushed to object storage as a persistent, compressed file. From then on that segment is searched through its index, not by brute force. A query therefore always blends two sources: sealed segments answered by their ANN index, and growing segments answered by exact scan, with results merged. Fresh and indexed data coexist because they live in different segment states.
Flush, index, and compaction: the lifecycle
Follow one vector through its life. It arrives at a proxy, gets a timestamp, and is appended to the log broker. A data node consuming that stream writes it into a growing segment and, on a flush, persists the raw segment to object storage for durability. When the segment seals, an index node reads it back, builds the ANN index, and writes that index file to storage too. A query node is told to load the sealed segment and its index into memory so it can serve searches.
Streaming ingestion creates a mess of small segments, so a background compaction merges small sealed segments into larger ones and physically purges rows marked deleted. This matters because deletes in Milvus are logical — a delete appends a tombstone rather than editing an immutable segment — so without compaction, deleted vectors keep costing memory and can still surface pre-filter. The lifecycle is thus a pipeline of specialised nodes handing a segment along: data node ingests and flushes, index node indexes, query node serves, compactor tidies.
Query nodes vs data nodes: splitting read from write
Because the roles are separate processes, Milvus scales the read path and the write path independently — the single most practical payoff of the disaggregated design. Data nodes own ingestion: they consume the write log, assemble growing segments, flush them, and cooperate on compaction. They are throughput machines, and a spiky bulk-import workload scales by adding data nodes without disturbing anyone who is querying.
Query nodes own search: they subscribe to the DML stream for growing segments, load sealed segments from object storage, hold vectors and indexes in RAM, and compute nearest neighbours. A read-heavy RAG service that must serve thousands of queries per second scales by adding query nodes and letting the coordinator rebalance segments across them. Neither tier holds unique state — the truth is in object storage and the log — so a query node can be killed and rebuilt by reloading its assigned segments. This clean separation is exactly what an in-process store cannot offer, and it is the reason Milvus targets the billion-scale, high-QPS end of the market.
The problem the index solves
Every vector database exists to avoid the exact nearest-neighbour scan, whose cost is O(N · d) multiply-adds per query — N vectors of dimension d. At a billion 768-dim vectors that is roughly 7.7 × 10^11 operations per query, hopelessly slow. Approximate nearest neighbour (ANN) search trades a provably-best answer for a near-best one obtained orders of magnitude faster, and Milvus offers three main families with genuinely different shapes.
The choice among them is a three-way tension between recall (fraction of true neighbours found), latency/QPS, and memory. No index wins all three; picking one is really deciding which resource you have most of. Milvus builds whichever you configure once per sealed segment, so the index type is a collection-level property fixed against your data size and hardware budget. The next sections take the three in turn.
IVF: partition the space, then probe
IVF (inverted file) clusters the vectors once with k-means into nlist cells, each with a centroid. To search, you compute the query’s distance to all nlist centroids, pick the nprobe nearest cells, and scan only the vectors inside them — ignoring the rest of the space entirely.
exact scan: cost = N * d
IVF search: cost ≈ (nlist + nprobe * N/nlist) * d
N=1e8, d=128, nlist=4096, nprobe=32:
centroids : 4096 * 128 ≈ 5.2e5
cells : 32 * (1e8/4096) * 128 ≈ 1.0e8
vs exact : 1e8 * 128 = 1.28e10 (~120x less)nprobe is the recall dial: probe one cell and you are fast but miss neighbours that fell just across a cell boundary; probe more and recall climbs toward exact as latency rises. Variants compress the payload — IVF_SQ8 scalar-quantizes floats to bytes (4× smaller), IVF_PQ uses product quantization for far more — trading a little recall for large memory savings. IVF is a strong default: modest memory, tunable, and it copes with the billion scale quantization is built for.
HNSW: the in-memory navigable graph
HNSW abandons partitioning for a layered proximity graph — a skip list generalised to many dimensions. Each vector is a node; upper layers are sparse long-range links, layer 0 holds everyone. A search enters at the top, greedily walks toward the query, drops a layer when it cannot improve, and refines at the bottom.
expected layers ≈ log_M(N)
search cost ≈ O(ef * log N * d)
N=1e7, M=16 → log_16(1e7) ≈ 5.8 layersTwo knobs govern it. M is the edges per node — more edges, better recall, permanently larger graph. ef (search) is the beam width at query time; raising it explores more of the graph for higher recall at higher latency, and it must exceed the requested k. HNSW delivers the best recall-at-latency of the three and excels at high QPS, but it pays in RAM: the full float32 vectors plus the graph links all live in memory, and the graph does not compress. It is the pick when latency and recall dominate and the working set fits in memory.
DiskANN: when the index will not fit in RAM
HNSW’s memory appetite becomes the wall at the billion scale: a billion 768-dim float32 vectors alone are about 3 TB, before graph links. DiskANN is Milvus’s answer for that regime. It builds a single Vamana graph designed to live on SSD, keeping only compressed vectors (via product quantization) and a little navigation state in RAM while the full-precision vectors and adjacency lists stay on disk.
A search walks the graph, using the in-memory PQ codes to steer cheaply and issuing SSD reads only to fetch the exact vectors of promising candidates for re-ranking. The cost model shifts from multiply-adds to random I/O: performance is bounded by how many SSD page reads a query needs, which is why DiskANN wants fast NVMe and is engineered to minimise reads per query. The trade is latency — a disk hop is far slower than a RAM lookup — in exchange for indexing datasets many times larger than memory at a fraction of the RAM cost. When your vectors dwarf your RAM budget, DiskANN is the only one of the three that fits.
Choosing an index
The three families map cleanly onto which resource is scarce:
| Index | Lives in | Strength | Reach for it when |
|---|---|---|---|
FLAT | RAM | Exact, 100% recall | Small collections; a recall baseline |
IVF_FLAT / SQ8 | RAM | Balanced, tunable, modest memory | A sensible default at scale |
IVF_PQ | RAM (compressed) | Big memory savings | RAM is tight, some recall is spendable |
HNSW | RAM | Best recall-at-latency, high QPS | Latency matters and it fits in memory |
DiskANN | SSD + RAM | Huge datasets on little RAM | Vectors dwarf your memory budget |
The honest way to choose is empirical: fix a target recall (say 0.95), then compare the QPS and memory each index needs to hit it on your data and hardware. Recall against the tuning knob (nprobe, ef) is steeply concave — it climbs fast then flattens — so measure against a FLAT baseline rather than guessing.
Consistency levels and the timestamp machinery
Because writes flow through a log that different components consume at different moments, Milvus cannot pretend every read sees every prior write for free — it exposes the trade explicitly as four consistency levels. Every operation carries a hybrid timestamp issued by the coordinator (a TSO, blending physical time and a logical counter). A query node knows its service time: the timestamp up to which it has fully consumed the log. A search carries a guarantee timestamp (GuaranteeTs), and the node waits until its service time reaches that value before answering.
The level simply chooses GuaranteeTs. Strong sets it to the latest issued timestamp, so the query blocks until every earlier write is visible — freshest, slowest. Bounded (the default) sets it slightly in the past, tolerating a small staleness window in exchange for not waiting. Session guarantees you read your own writes by using your last write’s timestamp. Eventually sets it to zero — never wait, read whatever is loaded. Picking a level is picking a point on the freshness-versus-latency curve, and the default deliberately shaves tail latency by allowing seconds of staleness.
Pitfalls that follow from the design
A few failure modes fall straight out of the machinery above:
| Symptom | Cause | Fix |
|---|---|---|
| ‘Missing’ just-written data | Bounded/Eventually staleness window | Use Strong or Session for read-after-write |
| Search errors or empties | Segments not loaded into query nodes | Load the collection; watch query-node RAM |
| Memory keeps climbing after deletes | Deletes are tombstones until compaction | Let compaction run; do not over-delete |
| Recall lower than a benchmark | nprobe/ef too low for your data | Tune against a FLAT baseline |
The connective tissue is that Milvus makes storage cheap and durable but makes memory and freshness the things you actively manage. A collection is only searchable once its segments are loaded into query-node RAM, staleness is a knob you chose whether you meant to or not, and deleted vectors linger until a background job reclaims them. Treat query-node memory and the consistency level as first-class decisions, not defaults you inherit.
IVF partitions and probes for a tunable, memory-modest default, HNSW gives the best recall-at-latency when the graph fits in RAM, and DiskANN puts a Vamana graph on SSD when the vectors dwarf memory. Freshness is an explicit dial: four consistency levels set a guarantee timestamp that trades staleness for latency. Manage query-node memory and the consistency level deliberately, and tune recall against a FLAT baseline rather than by intuition.