arXiv study finds MoE models overfit faster with repeated data, despite some remedies
Data repetition harms mixture-of-experts more than dense models; regularization helps but not fully.
- Publication
- arXiv
- Stage
- Preprint
- What we read
- Summary of the abstract
- Authors
- Atindra Jha, Margaret Li, Jure Leskovec, Percy Liang, Luke Zettlemoyer
- Universities and research institutions
- Not yet supplied in verified metadata; the Brief does not guess.
What the paper reports
Researchers tested MoE models from 80M to 1B active parameters with varying repetition rates. They found MoEs degrade more rapidly under data repetition, especially as sparsity increases, and performance falls below dense models under heavy repetition. Some regularization like dropout mitigates, enabling MoEs to beat dense models when data is repeated beyond certain thresholds, but not in all-unique training.
Why it matters
The result highlights limits of sparsity-driven efficiency when data is reused; benefits and risks depend on data strategy and regularization, not a universal win.
In this abstract-only study, researchers examine how repeating human-written text in training data affects mixture-of-experts (MoE) models compared with dense architectures. Across 80M–1B active parameters, MoEs show quicker degradation as repetition increases, with the gap widening as model sparsity grows. The work notes that strong masking-based regularization can rescue some performance, allowing MoEs to outperform dense models even when repetition is heavy, but never fully match all-unique data.
A secondary finding points to internal mechanisms: MoE routing stabilizes early in training, and expert specialization links to overfitting to repeated data. The authors emphasize that data strategy,what to repeat and how,will shape outcomes, and that no blanket guarantee applies to all users or tasks. The study is abstract-only and focuses on the interaction between sparsity and repetition, not real-world deployment guarantees.
What this does not tell us
Abstract-only scope and preprint status; no peer review reported here; findings are limited to MoE sparsity and data repetition in the specified experimental setup.
Original sources · 1
- Data Scarcity and Model Sparsity: Mixtures-of-Experts Overfit More to Repeated Data ↗arXiv · 2026-09-10
Check the original paper for its authors, methods, version and access terms.
What is your take?
Ask a question, add useful context or share a different perspective. Keep the conversation respectful and grounded.