What are embeddings?

From an ID to a position

A token ID is only an index, like a row number in a database. The model uses that ID to retrieve an embedding: a list of learned numbers. During training, those numbers move until they become useful for the model's prediction objective.

For a vocabulary embedding, one row might represent one token. For search, a separate embedding model can turn an entire sentence or document into one vector. Images, audio, users, and products can also be embedded. The input type and the training objective define what the geometry means.

flowchart LR
  X[Item: token, sentence, image] --> E[Embedding model or lookup]
  E --> V[Vector: many coordinates]
  V --> C[Compare, retrieve, or feed a network]

Real vectors often have hundreds or thousands of dimensions. A diagram with two axes is only a projection: a lossy view made for humans.

Interactive maps can make nearest neighbours, cosine similarity, analogies, and projections tangible. They are teaching instruments, not validation: a pleasing cluster can hide bias, collapse distinctions, or reflect only the projection rather than the full vector space.

Similarity is a chosen measurement

Cosine similarity compares the angle between two vectors. A value near 1 means they point in a similar direction, 0 means roughly unrelated directions, and -1 means opposite directions. Dot product and Euclidean distance are other common choices. The correct metric depends on how the model was trained and how the vectors were normalized.

Nearby vectors are not guaranteed to be synonyms. They may share topic, syntax, usage, visual appearance, user behaviour, or another property rewarded by training. An embedding space is a learned engineering artifact, not a universal map of meaning.

Where engineers use embeddings

In a transformer, token embeddings are the starting representation that later layers update using context. In semantic search, query and document embeddings support approximate nearest-neighbour retrieval. In a recommendation system, learned vectors can represent users and items. In clustering or duplicate detection, distance provides a useful signal but rarely a complete product decision.

Retrieval systems commonly combine embedding search with keyword search, metadata filters, authorization, and reranking. The vector index does not know which documents a user is allowed to read, whether a document is current, or whether a result actually answers the question.

Geometry does not certify meaning

An embedding is not a human-readable explanation of an item. One coordinate usually does not have a stable name such as “technical” or “positive.” The information is distributed across dimensions and depends on the model version.

Similarity is not truth, causality, safety, or permission. Two nearby passages can repeat the same error. Two relevant passages can be far apart because the embedding model does not cover their language or domain. Vectors produced by different model versions should not be mixed unless compatibility has been demonstrated.

Evaluate the downstream decision

Start with the downstream decision, not a pretty two-dimensional plot. For retrieval, create versioned queries with relevant, irrelevant, hard-negative, multilingual, and permission-sensitive documents. Measure recall at the candidate stage, ranking quality after reranking, latency, index freshness, and final task outcomes.

Slice results by language, domain, input length, rare terminology, spelling noise, and time. Test invariances you expect—such as harmless punctuation changes—and distinctions the system must preserve, such as negation, product versions, or access-control boundaries.

Record the embedding model, tokenizer, normalization, vector dimension, distance metric, and index build. When any of these changes, rebuild or migrate the index and run the frozen suite before comparing production metrics.