Vector Databases

Vector databases arrived alongside the current wave of AI tooling and picked up a slightly mystical reputation on the way. They are not mysterious. A vector database stores lists of numbers and answers one question quickly: which of these are closest to this one.

Everything else, the index structures, the distance metrics, the quantisation schemes, exists to make that one question fast at scale without giving up too much accuracy. Here is what is going on underneath, and how to pick one without buying more machinery than the problem needs.

What a vector is here

An embedding model takes a piece of text, or an image, or audio, and produces a fixed length list of numbers. Several hundred to a couple of thousand of them, depending on the model. That list is the vector.

The useful property is that inputs with similar meaning produce vectors that sit near each other in that space. "How do I reset my password" and "I forgot my login details" share almost no words, so keyword search connects them poorly. Their vectors land close together, so similarity search finds one from the other.

That is the entire premise. The database's job is to hold millions of these and find the nearest handful quickly.

Why you cannot just compare everything

Finding the nearest vector by brute force is simple: compute the distance to every stored vector and sort. It is also exact, which is worth remembering, because people forget that brute force is an option.

The cost is linear. Ten thousand vectors of 1024 dimensions is around ten million multiplications per query, which a modern machine does without noticing. Ten million vectors is ten billion, which it very much notices.

So beyond a certain size you switch to approximate nearest neighbour search. Approximate is the important word. These indexes trade a small amount of accuracy for a very large amount of speed, and the accuracy you give up is measured as recall: of the true nearest neighbours, what fraction did the index actually return.

Recall of 0.95 means one result in twenty that should have appeared did not. For a retrieval system feeding a language model, that is usually an acceptable trade. For an exact deduplication check, it is not. Know which one you are building.

Distance metrics

Three are in common use, and the choice is mostly dictated by your embedding model rather than by preference.

Cosine similarity measures the angle between two vectors and ignores their length. This is the default for text embeddings, because what you care about is direction of meaning rather than magnitude.

Dot product is cosine similarity's faster cousin. If your vectors are normalised to unit length, which most embedding models do for you, dot product and cosine give identical rankings and the dot product is cheaper to compute. Check whether your model normalises; if it does, use dot product.

Euclidean distance is straight line distance in the space. It shows up more in image and audio work than in text.

The practical rule: use whatever metric your embedding model was trained with. Using a different one will not error, it will just quietly return worse results, which is a much harder problem to notice.

The index types

Two dominate, and understanding the difference explains most of the configuration options you will meet.

HNSW

Hierarchical navigable small world graphs. The most widely used index and the default in most stores.

The structure is a set of layers. The top layer holds a few vectors connected by long range links. Each layer down holds more vectors with shorter links, until the bottom layer holds everything. A search starts at the top, greedily walks towards the query, drops a layer, walks again, and repeats. It is a motorway network: long hops to get near, then local roads to arrive.

Two parameters matter.

M is how many connections each node keeps. Higher means better recall and more memory. Between 16 and 64 covers most cases; 16 is a sensible default.

ef_construction is how hard the index works when inserting. Higher builds a better graph and takes longer. This is set once at build time.

Then at query time, ef_search controls how many candidates the search examines. This is the dial you actually tune in production, because it trades recall against latency per query and can be changed without rebuilding anything. Turn it up when results look thin, down when queries are slow.

HNSW's characteristics: excellent recall, fast queries, high memory use, and slow to build. The memory is the thing that surprises people. The graph lives in RAM, and for high dimensional vectors the index can approach the size of the vectors themselves.

IVF

Inverted file index. The vectors are clustered during a training step, each cluster gets a centroid, and every vector is filed under its nearest centroid. A search compares the query to the centroids, picks the closest few clusters, and searches only inside those.

nlist is the number of clusters, often set around the square root of your vector count. nprobe is how many clusters to search at query time, and it is the recall dial: searching more clusters finds more of the true neighbours and takes longer.

IVF uses far less memory than HNSW and builds faster. Its weakness is the boundary problem. A vector sitting near the edge of a cluster may be a true nearest neighbour while living in a cluster the search did not open. Raising nprobe mitigates this at the cost of speed.

IVF also needs training data before it can build, so it is awkward for a collection that starts empty and grows.

Which to use

HNSW unless memory is the binding constraint. It is the default in most stores for good reason. Reach for IVF, usually combined with quantisation, when the dataset is large enough that keeping an HNSW graph in RAM is the expensive part of your bill.

Quantisation

The lever for cutting memory, and worth knowing because it is what makes large collections affordable.

A vector is normally stored as 32 bit floats. A thousand dimensions is four kilobytes per vector, so ten million vectors is forty gigabytes before any index overhead.

Scalar quantisation stores each number as an 8 bit integer instead. Four times smaller, with a small accuracy loss. This is close to free and worth turning on for most large collections.

Binary quantisation reduces each dimension to a single bit. Thirty two times smaller, and distance comparisons become bitwise operations, which are extremely fast. Accuracy drops considerably on its own, so it is used as a first pass: search the binary index broadly, then rescore the top candidates against the full precision vectors. The combination gives most of the accuracy at a fraction of the memory.

Product quantisation splits the vector into segments and encodes each against a learned codebook. More compression than scalar, more accuracy loss, and usually paired with IVF.

The rescoring pattern generalises. Search a compressed index widely, then rerank a small candidate set precisely. It shows up everywhere in this field for the same reason: cheap and broad, then expensive and narrow.

Filtering, and why it is harder than it looks

Most real queries are not purely semantic. You want the nearest vectors among documents this user can see, or from the current policy version, or in this department.

There are three ways to do it, and the difference matters.

Pre filtering applies the condition first, then searches the remainder. Exact, but it can defeat the index: if the filter leaves a small subset scattered across the graph, the search structure no longer helps and you are close to brute force.

Post filtering searches first, then discards non matching results. Fast, and it has an unpleasant failure mode. Ask for ten results, get ten from the index, discard nine that fail the filter, and return one. The system looks like it found nothing when the matches existed further down the ranking.

Filtered search applies the condition during graph traversal, skipping nodes that fail it while still navigating through them. This is what the better stores do, and it is the main technical difference between a mature vector database and a bolted on similarity function.

If your application filters heavily, and most do once permissions enter the picture, test this specifically. It is where implementations diverge most.

Choosing a store

pgvector if you already run Postgres. This is the right answer more often than the discourse suggests. Your vectors sit alongside your relational data, so filtering by user, tenant, status or date is an ordinary WHERE clause, and you get transactions, backups, replication and a tool your team already knows. It supports HNSW and IVF, and it is comfortable into the millions of vectors. The ceiling is real but higher than most projects reach.

Qdrant for a dedicated store with strong filtered search and straightforward operation. A good next step when Postgres stops fitting.

Weaviate when you want hybrid search, schema and modules in one system rather than assembled from parts.

Milvus for very large deployments where the distributed architecture earns its complexity. Overkill below serious scale.

Pinecone or another managed service when you would rather not operate any of it. You pay for that, and you accept your vectors living elsewhere.

The honest advice: start with pgvector if you have Postgres, or the simplest option you can otherwise run. Migrating vectors later is genuinely easy, because you have the source documents and re embedding is a batch job. Choosing a distributed store for a hundred thousand vectors is a cost you pay every day for a problem you may never have.

Operating one

The things that cause trouble in production, none of which are about search quality.

Memory is the constraint. HNSW indexes live in RAM. Estimate before you build: roughly the vector data plus graph overhead, and the overhead scales with M. Running out of memory on an index rebuild during a busy period is a bad way to learn this.

Deletions accumulate. Most implementations mark deleted vectors as tombstones rather than removing them from the graph, because removal is expensive. They still consume memory and are still traversed. A collection with heavy turnover needs periodic compaction, and that compaction is resource intensive. Schedule it.

Updates are usually delete plus insert. There is no cheap in place edit of an indexed vector. A workload that constantly revises documents behaves quite differently from one that mostly appends.

Changing the embedding model means re embedding everything. Vectors from different models are not comparable, so there is no gradual migration. Plan it as a full rebuild with a cutover, ideally building the new collection alongside the old and switching once it is verified.

Measure recall, not just latency. Take a sample of queries, compute exact results by brute force, and compare against what the index returns. Without that number you have no idea whether a configuration change helped or quietly degraded the system, because a lower recall index is faster and looks better on every dashboard you have.

When you do not need one

Worth saying plainly, because the tooling is easy to over adopt.

Below roughly a hundred thousand vectors, brute force in memory is fine. A NumPy matrix multiplication over fifty thousand vectors takes milliseconds, gives exact results, has no index to tune, no recall to measure and no service to operate.

If your search is genuinely keyword based, a full text index is better and cheaper. Semantic search is not universally superior; it is better at meaning and worse at exact terms, which is why serious systems run both and combine the rankings.

And if the corpus is small enough to fit in a model's context window, you may not need retrieval at all. Passing the documents directly is simpler, exact, and increasingly viable as context windows have grown. It costs more per request, so the arithmetic depends on your volume, but it is worth doing the sum before building an index.

Worth knowing

A vector database is a specialised index, not a magic component. The interesting decisions are ordinary engineering ones: how much accuracy you are willing to trade for speed, how much memory you are willing to buy, and whether the complexity is justified by the scale you actually have rather than the scale you imagine.

Get the embedding model and the chunking right first. Those determine whether the right answer is findable at all. The index only determines how quickly you find it.

If you are building semantic search over your own data, or you have a vector store that is slower or less accurate than it should be, talk to Eight Mile. RAG systems, AI assistants, backend APIs, cloud infrastructure and system architecture are the work we take on.