Neural networks receive numbers, not raw text. A tokenizer breaks text into reusable units called tokens. Tokens may be words, word pieces, punctuation, or bytes. The chosen vocabulary determines how compactly different languages and unusual strings are represented.
Each token maps to an integer ID. IDs are labels, not quantities: token nine is not larger or more meaningful than token three. A vocabulary table provides stable conversions between token strings and IDs.
An embedding table maps every token ID to a learned vector. Looking up a sequence of IDs produces a matrix with one row per sequence position and one column per model feature. Similar uses can develop related vectors during training.
Consider the text 'cats nap'. It may become three tokens including punctuation or spaces, depending on the tokenizer. Each token retrieves one vector. The resulting rows preserve token identity but still need position information to distinguish reordered sequences.
Unknown text is handled by smaller reusable pieces rather than requiring a vector for every possible word. Smaller pieces improve coverage but create longer sequences. Longer sequences cost more computation and may split meaningful units awkwardly.
Inspect token strings, IDs, sequence length, padding, and embedding shape on real examples. Try names, code, numbers, and multiple languages. Many apparent model failures begin with surprising tokenization or truncation before the transformer sees the content.