How does vector search work?

From documents to candidates

A vector-search system prepares material before the first query. It splits records into retrievable units, attaches identity and access metadata, produces one embedding per unit with a declared model, and places those vectors in an index.

At query time, the same compatible embedding model turns the request into a vector. The index finds nearby stored vectors under a metric such as cosine similarity, dot product, or Euclidean distance. The result is a candidate list—not yet an answer and not necessarily the final ranking.

flowchart LR
  D[Versioned documents] --> C[Chunks plus metadata]
  C --> E[Embedding model]
  E --> I[Vector index]
  Q[Query] --> QE[Same compatible model]
  QE --> I
  I --> K[Nearest candidates]
  K --> A[Authorization and filters]
  A --> R[Rerank or combine with keyword search]
  R --> X[Selected evidence]

Document identity, source location, version, tenant, permissions, timestamps, and transformation lineage must remain attached to the vector. The coordinates alone cannot reconstruct those contracts.

Exact search and approximate search

Exact nearest-neighbour search compares the query with every stored vector and returns the true nearest items under the chosen representation and metric. It is a valuable ground truth for smaller datasets, but its work grows with the collection.

Approximate nearest-neighbour indexes avoid many comparisons. They trade some recall for lower latency or memory. HNSW, for example, organizes vectors as layered proximity graphs: a search makes coarse moves in upper layers and finer moves near the query in lower layers. Product-quantization approaches compress vectors and search the compressed representation. Inverted indexes first narrow the search to selected partitions.

These are data-structure choices, not different meanings of “semantic.” Configuration controls construction time, memory, query latency, and the probability of retrieving the exact nearest items. A setting that works on one vector distribution may not transfer to another.

What a vector database adds

A vector index answers a proximity-search problem. A vector database or search service usually adds persistence, updates, deletion, replication, metadata filtering, namespaces, APIs, durability, and operational monitoring around one or more index types.

It still does not provide business authorization merely because it supports a filter expression. The application must bind every query to the current principal and prove that forbidden records cannot enter the candidate set or logs. Deletion must propagate to vectors, replicas, caches, and derived indexes.

Updates also create consistency decisions. A source document can change before its old vector is replaced. An embedding-model upgrade produces a different space; mixing old and new vectors is unsafe unless compatibility has been demonstrated. Re-embedding, dual indexes, cutover, and rollback need explicit versions.

Similarity is not relevance

An embedding captures relationships rewarded by its training objective. Nearness may reflect topic, style, syntax, or broad association while missing an exact product code, negation, date, or version. The closest vector can be irrelevant, stale, duplicated, or unauthorized.

Keyword search remains strong for exact identifiers and rare strings. Metadata filters express known constraints. Rerankers can compare the query with a smaller candidate set more precisely. Hybrid retrieval often works because these mechanisms fail differently.

Vector search is therefore one stage of RAG, not a synonym for it. RAG additionally owns evidence selection, context construction, generation, citations, and abstention.

Measure the operating point

Build a query set with known relevant records and important non-relevant records. Include exact identifiers, paraphrases, multilingual queries, near-duplicates, old versions, hard negatives, and permission boundaries. For approximate search, compare results with exact search on the same frozen vectors.

Measure recall at the candidate depth, final ranking quality, p50 and tail latency, memory, build time, update lag, and cost. Change one search parameter at a time and plot recall against latency rather than naming one configuration “fast.”

Then test the operational contracts: concurrent updates, interrupted builds, corrupt vectors, model-version mismatch, deletion, failover, empty filters, missing metadata, cross-tenant queries, and a relevant item just below the cutoff. A retrieval system is useful when it returns authorized evidence reliably under the real query distribution—not when a two-dimensional embedding plot looks persuasive.