What is a token?

A token is the smallest unit of text that a language model reads or writes. Before your prompt reaches the model, a tokenizer splits the input string into these pieces and maps each one to an integer id. The model only ever sees those ids, never raw characters.

Tokenization is not word-for-word. Frequent short words often become a single token, while rarer words split into fragments; punctuation and whitespace usually attach to neighboring pieces. Different model families use different tokenizers, so the same sentence can produce different token counts on different models.

Tokens have three practical consequences. Context windows are measured in tokens, so the combined size of your prompt and the model's reply has to fit inside that budget. API pricing is quoted per token, separately for input and output. And some text is disproportionately expensive: code, unusual spellings, and many non-English scripts expand into more tokens than their character count suggests.

Because counts are hard to eyeball, every major provider ships a tokenizer library or a counting endpoint. Measure a request before sending it: token math is the difference between a prompt that fits and one that is silently truncated.