Publication
arXiv
Stage
Preprint
What we read
Summary of the abstract
Authors
Ahmetcan Yavuz, Clara Meister, Tiago Pimentel
Universities and research institutions
Not yet supplied in verified metadata; the Brief does not guess.

What the paper reports

arXiv researchers introduced BottomUpLL and TopDownComp to map the 2x2 design space of tokenisers, then trained language models using these tokenisers across sizes and domains (English-only vs multilingual). They evaluated models on bits-per-byte and BLiMP tasks, finding that the search procedure is the dominant factor for efficiency, while performance on BLiMP shows no consistent link to design choice.

Why it matters

The study challenges assumptions about tokeniser design by showing search strategy may drive efficiency more than the chosen objective, though model performance effects are not uniform across tasks.

The researchers explain that two widely used tokenisers,BPE and UnigramLM,were examined in a structured way by separating the objective (compression vs. likelihood) from the search approach (bottom-up vs. top-down). By training models with BottomUpLL and TopDownComp across sizes and languages, they reveal how much tailors to search matters for efficiency.

The findings show that, across most settings, bottom-up, likelihood-based tokenisers achieve lower bits-per-byte, indicating a practical edge in resource use, even if not implying universal performance gains across all tests, such as BLiMP.

What this does not tell us

Abstract-only scope and preprint status; conclusions based on controlled design in an abstract-only paper, not a full peer-reviewed article.

Original sources · 1
  1. Objective vs. Search: Decomposing What Makes a Good Tokeniser ↗arXiv · 2026-09-16

Check the original paper for its authors, methods, version and access terms.