What is a token?

A token is the unit a language model actually reads and writes. Models don't see letters or words — before your prompt reaches the model, a tokenizer chops it into chunks from a fixed vocabulary and hands the model a list of numbers. Everything downstream is priced in this unit: the context window is a token budget, rate limits are tokens per minute, and your bill is dollars per million tokens.

The chunks come from byte-pair encoding (BPE): during training, the tokenizer-builder starts from raw bytes and repeatedly merges the most frequent adjacent pairs until it has a vocabulary of a chosen size — around 200,000 entries for OpenAI's current o200kencoding. Frequent sequences earn their own entry (" the" is one token, including its leading space); rare words get assembled from pieces. That's why tokenization might split into token + ization, and why a typo can double a word's token count — the misspelling isn't in the vocabulary.

Rules of thumb for English text that hold up well in practice:

• 1 token ≈ 4 characters ≈ ¾ of a word
• 1,000 words ≈ 1,300–1,500 tokens
• Code runs heavier than prose (indentation, brackets, camelCase), and languages like Chinese or Japanese can cost 1.5–3× more tokens per sentence than English

The catch: every vendor trains its own tokenizer, so the same text becomes different token counts on different models. A prompt that's 1,000 tokens on GPT-5.6 can be 1,400+ on a Claude model — which matters because the price sheets all say "per million tokens" as if a token were a standard unit. It isn't. That's the entire reason this site exists, and it's covered in depth in the effective price of frontier models.

Chat prompts carry one more hidden cost: the chat template. Your messages get wrapped in role-marker tokens (<|im_start|>system…) before the model sees them, so a "one-word prompt" is never one token. The comparison table applies each model's real template so the counts match what you'd be billed — click any row to see the template's special tokens highlighted.