Tokenization for LLM Systems (September 2026)¶
Reviewed 2026-09-03. Tokenization maps input text to the integer sequences a model consumes. Token counts, special tokens, truncation, and chat formatting are model-specific; never use a generic characters-per-token estimate for a budget or hard limit.
Tokenization Pipeline¶
| Stage | Purpose |
|---|---|
| Normalization | Applies configured transformations to raw input |
| Pre-tokenization | Splits text into candidate units |
| Model | Maps units/subwords to token IDs |
| Post-processing | Adds special tokens and sequence structure |
| Encoding | Holds IDs, offsets, masks, and related metadata |
The Hugging Face Tokenizers API documents this pipeline and the tokenizer components. Tokenizer API
Algorithm Families¶
| Family | Typical property | Use with care |
|---|---|---|
| BPE | Frequent merges create reusable subwords | Vocabulary and byte handling differ by implementation |
| WordPiece | Subword scoring with continuation conventions | Token representation is tokenizer-specific |
| Unigram / SentencePiece | Probabilistic subword vocabulary and raw-text handling | Whitespace and normalization policy matter |
| Character/byte fallback | Coverage for unusual text | Longer sequences can change cost and latency |
The algorithm name does not determine cost or context behavior. The actual tokenizer artifact, chat template, and request serialization do.
Count with the Exact Tokenizer¶
This Python example uses a named tokenizer artifact and returns the exact IDs for that artifact.
from tokenizers import Tokenizer
tokenizer = Tokenizer.from_pretrained("bert-base-uncased")
encoding = tokenizer.encode("A token budget must use the deployment tokenizer.")
print("token_count=", len(encoding.ids))
print("tokens=", encoding.tokens)
For hosted APIs, use the provider documented tokenizer/counting method for the exact model and request format. Include system messages, tool definitions, retrieved context, attachments, and expected output reservation in the budget.
Context Budget¶
context_limit
- fixed instructions
- tool schemas
- conversation state
- retrieved evidence
- current user input
- output reservation
= remaining input budget
Do not truncate silently. Define which content can be summarized, retrieved again, rejected, or held for a smaller task.
Unicode and Structured Data¶
- Different languages, code, JSON, URLs, and base64 payloads can have very different token densities.
- A visible character sequence can normalize or split differently between tokenizers.
- Tool definitions and large tool results can consume context before the user input is processed.
- Multimodal request accounting is provider-specific; use the provider current guidance instead of treating media as text tokens.
Gotchas¶
- Issue: Estimating costs with a fixed characters-per-token rule. The estimate can be wrong for non-English text, code, or structured data. Fix: count with the exact deployment tokenizer and full request serialization.
- Issue: Using the base-text tokenizer but omitting chat/tool formatting. The request can exceed the real context limit. Fix: account for system instructions, tool schemas, retrieved chunks, and output reserve.
- Issue: Truncating the newest evidence by accident. A simplistic slice can preserve irrelevant history and lose the task. Fix: define a state-aware truncation or retrieval policy.
- Issue: Changing tokenizer artifacts without evaluation. Token boundaries can affect retrieval, prompt length, and model behavior. Fix: version the tokenizer with the model configuration.