Beginner
What is a Token?
The basic unit of text that language models actually read and write.
Models don't see words or characters directly — they see tokens, chunks of text produced by a tokenizer. A token might be a whole common word (' the'), part of a word (' un' + 'believ' + 'able'), or a single character for rare text.
Roughly speaking, 1 token ≈ ¾ of a word in English, so 100 tokens ≈ 75 words. Tokenization affects cost (APIs charge per token), speed, and how well a model handles different languages.
Token limits matter: a model's context window is measured in tokens, and everything you send plus everything it generates counts against that budget.
Key points
- Tokens are text chunks, not words
- ~¾ word per token in English
- APIs price by the token
- Context windows are measured in tokens