The smallest unit of text a language model reads or writes, produced by a tokenizer. Context windows and API pricing are measured in tokens.
Tokens are the smallest units of text a language model reads or writes, and every context limit and price is measured in them.