Bottom-up tokenisers save space but not for every language
A study shows how tokeniser design affects language models. Bottom-up approaches usually use fewer bits per byte but don’t reliably improve grammar judgments.
- Publication
- arXiv
- Stage
- Preprint
- What we read
- Summary of the paper
- Authors
- Ahmetcan Yavuz, Clara Meister, Tiago Pimentel
- Universities and research institutions
- Not yet supplied in verified metadata; the Brief does not guess.
What they did and found
Researchers compared two tokeniser designs: one that grows tokens by joining small text parts (bottom-up), and one that trims a large token set (top-down). Bottom-up often cuts data size, but does not reliably boost grammar judgments across languages.
Why it matters
For business AI work, results vary: smaller size can bring trade-offs in language handling, and grammar tests aren’t guaranteed to improve.
Two approaches were tested: bottom-up merges small text parts into tokens, while top-down trims tokens from a large pool. Bottom-up reduced data size more, but language quality did not consistently improve.
Practically, you might save storage or speed up work with bottom-up methods, but test language results for your own tasks and languages.
Original sources · 1
- Objective vs. Search: Decomposing What Makes a Good Tokeniser ↗arXiv · 2026-09-16
Check the original paper for its authors, methods, version and access terms.
What remains uncertain
Findings may not hold for models bigger than one billion parameters or languages not included in the tests.
What is your take?
Ask a question, add useful context or share a different perspective. Keep the conversation respectful and grounded.