Chroma Math + Architecture

The core idea: an embedded vector store

Most vector databases are servers you deploy, connect to over the network, and scale on their own machines. Chroma’s default mode is the opposite: a PersistentClient opens a directory on local disk and runs the entire engine — index, storage, query planner — inside your application’s own process. There is no wire hop between your code and the index; a query is a function call.

The right mental model is SQLite for vectors. Just as SQLite trades a database server for a single-file library you embed, Chroma trades a vector cluster for an in-process library that persists to a folder. The payoff is enormous for its target workloads — prototypes, notebooks, single-node RAG services, desktop and edge apps — because you remove an entire network tier. The cost is the one SQLite pays: a single process owns the data, so concurrency and horizontal scale are bounded by that process rather than a fleet. Chroma also ships a client/server mode and hosted Chroma Cloud for when you outgrow the embedded shape, but the embedded case is what makes Chroma Chroma.

hnswlib: the engine Chroma stands on

Every vector store is a way to avoid an exact nearest-neighbour scan, whose cost is O(N * d) multiply-adds per query over N embeddings of dimension d — a few milliseconds at 200k vectors, but hundreds of milliseconds once N reaches the millions. Chroma does not implement its own escape from that wall: for each collection it builds and queries an HNSW graph through hnswlib, the compact C++ library that popularised Hierarchical Navigable Small World graphs. Chroma’s job is the surrounding system — collections, persistence, filtering, the API — while the nearest-neighbour math lives in that battle-tested dependency.

HNSW is a skip list generalised to many dimensions. Every vector is a node in a proximity graph, and each node is assigned a maximum layer from an exponential distribution, so upper layers are sparse long-range sketches and layer 0 holds everything:

P(level ≥ l) ≈ M^(-l)     → layer l holds ≈ N / M^l
expected layers    ≈ log_M(N)
search cost        ≈ O(log N * d)

A query enters at the top, greedily walks toward the query vector, drops a layer when it can no longer improve, and refines at the bottom. With M = 16 and N = 2,000,000, that is log_16(2e6) ≈ 5.2 layers and a few hundred node visits instead of two million — a logarithmic descent, at the price of occasionally returning a neighbour that is excellent rather than provably best.

Collections: the unit of indexing

A Chroma collection is the table-like abstraction you actually work with: you add documents, embeddings, and metadata to a named collection and query it. Under the hood each collection owns its own HNSW index and its own storage segment — they are not slices of one shared graph. That has direct consequences for how you model data and how you pay for it.

Because indexes are per-collection, a search never traverses vectors outside the collection you query, and the graph’s size is exactly that collection’s N. Splitting data into many small collections gives cheap, hard isolation — per-tenant separation with no filter needed — but the total resident memory is the sum of every collection’s index, and a query cannot span collections. One large collection with a metadata field wins when you search the whole corpus and occasionally narrow; many collections win when the partitions are genuinely independent. The distance metric, too, is fixed per collection at creation time.

The three distance metrics

A collection declares its space through hnsw:space, one of l2 (the default), cosine, or ip (inner product). They are less independent than they look. Expand squared Euclidean distance:

||q - v||^2 = ||q||^2 - 2 (q · v) + ||v||^2

cos(q, v)   = (q · v) / (||q|| * ||v||)

if ||q|| = ||v|| = 1:
    ||q - v||^2 = 2 - 2 (q · v) = 2 - 2 cos(q, v)

On unit-normalised vectors, ranking by l2, cosine, and ip gives identical orderings — the metrics differ only by a monotone transform. The distinction bites when norms vary: raw ip rewards long vectors, so a document can rank highly for being big rather than relevant, while cosine divides magnitude out. Note too that Chroma’s l2 returns the squared distance and its cosine field is a distance (1 - cos), not a similarity — a routine source of confusion when you sort or threshold. Pick the metric your embedding model was trained under and normalise at write time.

The knobs: M, construction_ef, search_ef

Chroma surfaces hnswlib’s parameters as collection metadata; each buys recall with a different currency.

The asymmetry is the whole point. construction_ef is a one-off tax at insert time that improves graph quality forever, so generosity there is nearly free; search_ef is paid on every query and must exceed the requested k. Recall against search_ef is steeply concave — it climbs fast, then flattens — so tune it against an exact scan, not by intuition. Raising M helps on hard, high-dimensional datasets but permanently enlarges the graph — reach for it last.

The brute-force buffer, and why adds feel instant

Building an HNSW graph is not free: each insert is itself a small graph search, so rebuilding on every add would make ingestion crawl. Instead Chroma keeps a small brute-force segment in front of the HNSW index. New vectors land there first and are searched exactly — cheap while the buffer is small — and are periodically batched into the graph in the background.

Two settings govern the rhythm: hnsw:batch_size, how many vectors accumulate before a merge, and hnsw:sync_threshold, how often the graph is persisted to disk. The effect is that a query answers from two sources — the HNSW graph for settled vectors and the brute-force buffer for recent ones — and Chroma merges the results. This is why freshly added items are immediately searchable with exact accuracy, and why bulk ingestion is smooth rather than spiky. The knock-on cost is that a large buffer means more exact scanning per query.

Persistence and metadata: the SQLite half

Chroma splits storage along a clean seam. The vectors and the HNSW graph live in a binary index segment on disk, loaded into memory to serve search. The documents, ids, and metadata live in a SQLite database in the same persistence directory. That split is why the embedded story holds together: SQLite gives durable, transactional, queryable metadata storage without a server, matching Chroma’s in-process ambition.

It also explains the performance shape. A get by id or a document listing hits SQLite, not the graph; a vector query walks the graph and then joins back to SQLite to hydrate the matching documents and metadata. That separation makes the where filter, which straddles both, the most interesting part of the query path.

Filtering: pre-filter through SQLite

Real queries are rarely pure vector queries: you want the nearest neighbours among documents with source = "docs" and year ≥ 2024. Chroma expresses this with a where clause on metadata (and where_document on text), and the order of operations decides whether the result is trustworthy. Let s be the selectivity, the fraction passing the filter.

post-filter: take ANN top-C, then drop non-matches
             survivors = C * s   →  to return k you need C ≈ k / s
             k=10, s=0.001  →  C ≈ 10,000

pre-filter:  where clause in SQLite → allowed id set,
             then constrain the HNSW walk to those ids

Chroma resolves the where clause against SQLite first and restricts the search to the surviving ids, so it does not silently return three documents when you asked for ten — the failure mode of naive post-filtering. The catch is structural: the HNSW graph was built over all N points, so on a highly selective filter the reachable subgraph fragments and the greedy walk stalls far from the true neighbours. When s is small, let an exact comparison over the small survivor set do the work instead.

A RAM cost model you can budget with

Because the HNSW index is resident in memory, sizing Chroma is one question: how many bytes does the index occupy? hnswlib stores each full float32 vector inline with its neighbour links, so:

vector bytes = N * d * 4                  (float32)
graph bytes  ≈ N * 2M * 4              (layer-0 ids)
             * (1 + 1/(M-1))          upper layers

N = 1,000,000, d = 768, M = 16:
  vectors = 1e6 * 768 * 4          = 3.07 GB
  graph   = 1e6 * 32 * 4 * 1.07    = 0.14 GB
  total                            ≈ 3.2 GB

Two lessons fall out. First, the vectors dominate the graph by roughly twenty to one, so dimension d and count N drive the bill far more than M. Second — the crucial embedded-specific point — this memory lives in your application’s process: a 3 GB index is a 3 GB addition to your service’s resident set, not a separate box’s problem. Chroma’s embedded core does not quantize vectors, so this float32 figure is the number to plan against.

Pitfalls worth internalising

A short list of failures that follow directly from the machinery above:

The unifying theme is that approximate, in-process search fails quietly — no exception when recall drifts or when the index outgrows the box. So keep a few hundred labelled queries, check recall against an exact baseline periodically, and watch your process’s resident memory as a first-class metric alongside latency.