Transformers' distance generalization varies by positional encoding, NoPE often helps
Study shows distance generalization in transformers depends on positional encoding; NoPE can outperform explicit schemes like RoPE/ALiBi, with transfer learning yielding mixed results.
- Publication
- arXiv
- Stage
- Preprint
- What we read
- Summary of the paper
- Authors
- Daniel Henrik Nevermann, Claudius Gros
- Universities and research institutions
- Not yet supplied in verified metadata; the Brief does not guess.
What they did and found
Researchers trained decoder-only transformers on synthetic delay copy tasks with fixed context length, comparing ALiBi, RoPE, and NoPE. They found NoPE often yields better distance generalization, RoPE underperforms, and data diversity yields diminishing returns.
Why it matters
Positional encoding choices shape how AI handles token spacing, affecting robustness and transfer learning outcomes in real tasks.
In these synthetic delay-copy tasks, models must recall earlier tokens after a delay within a fixed context. The study shows NoPE often generalizes better to unseen delays, while RoPE tends to delay performance and ALiBi varies.
Practitioners should weigh model size and encoding strategy, noting that transfer learning can help in some cases but hinder in others, depending on task distance and diversity.
Original sources · 1
Check the original paper for its authors, methods, version and access terms.
What remains uncertain
Results come from synthetic tasks and small models; real pretrained models may behave differently and transfer effects depend on task specifics.
What is your take?
Ask a question, add useful context or share a different perspective. Keep the conversation respectful and grounded.