How can a categorical ID become something a neural network can transform?

Identity plus place becomes model input

A token ID is only an address. The model uses it to look up a trainable vector that gives the token a useful starting representation. Identity alone cannot distinguish “dog bites person” from “person bites dog,” so the model also needs position information. The token signal and position signal are combined before or within attention, depending on the architecture. This is still only a starting state, not the rich contextual meaning formed by later blocks. The same token can begin with the same lookup vector in two sentences and develop different internal states as it gathers different context. The observatory uses real static GloVe word vectors to make neighborhoods, directions, and analogy arithmetic inspectable; it demonstrates learned geometry without pretending those rows are Bonsai’s private token weights.

A token ID selects a learned vector; position makes word order available.