arXiv AI
Jul 2
CausalMix: Data Mixture as Causal Inference for Language Model Training
arXiv:2607. 01104v1 Announce Type: cross Abstract: In Large Language Model (LLM) training, data mixing plays a pivotal role in determining model performance.
TANDEM: Bi-Level Data Mixture Optimization with Twin Networks
arXiv:2606. 04401v1 Announce Type: new Abstract: The capabilities of large language models (LLMs) significantly depend on training data drawn from various domains.
Related stories
arXiv Machine Learning
Jul 14
Domain-Aware Scaling Laws Uncover Data Synergy
arXiv:2607. 11052v1 Announce Type: new Abstract: Machine learning progress is often attributed to scaling model size and dataset volume, yet the composition of data can be just as consequential.
arXiv Machine Learning
Oct 3
SLIM: Simplex-Lattice Interpolation Merging
The paper introduces Simplex-Lattice Interpolation Merging (SLIM), a method that builds a quadratic surrogate model of aggregate performance over merging coefficients for large language models using a classical mixture design. By evaluating only individual experts and equal-weight pairs, SLIM identifies the surrogate with the fewest measurements needed and then optimizes it without further target-metric evaluations. Experiments on two model architectures show that SLIM accurately predicts unseen multi-expert mixtures and achieves competitive merge performance under limited evaluation budgets, outperforming random designs and regularized fitting.
arXiv Computation and Language
Oct 1
DataFlex: A Unified Framework for Data-Centric Dynamic Training of Large Language Models
arXiv:2603.26164v2 Announce Type: replace-cross Abstract: Data-centric training has emerged as a promising direction for improving large language models (LLMs) by optimizing not only model parameters...
arXiv AI
Aug 26
Data Mixing as Mixture Experiment: Response Surface Methodology and Optimal Design for Large Language Model Pretraining
The article proposes treating large language model (LLM) data mixing as a classical mixture experiment, where data domains are components, token shares are proportions, and proxy-training runs serve as design points. Using sparse second‑order Scheffé response‑surface models, the authors construct model‑robust Σ‑optimal designs that efficiently identify optimal data mixtures and reveal strong interaction effects, especially between weak domains and web‑derived text. Empirical results on RegMix show that these designs recover mixture rankings while reducing proxy runs by about 25%, demonstrating that data mixing can be optimized through experimental design rather than solely prediction.
arXiv AI
Oct 7
Cross-Lingual Activation Steering for Multilingual Language Models
arXiv:2601.16390v2 Announce Type: replace-cross Abstract: Large language models exhibit strong multilingual capabilities, yet significant performance gaps persist between dominant and non-dominant la...
arXiv AI
Jul 28
DomainPilot: Domain-Level Loss-Guided Two-Stage Data Mixture Optimization for Efficient Language Model Fine-Tuning
arXiv:2607. 22769v1 Announce Type: cross Abstract: The training efficacy of large language models (LLMs) is fundamentally constrained by the quality and composition of training data.
arXiv Computation and Language
Aug 21
EvoSelect: Data-Efficient LLM Evolution for Targeted Task Adaptation
arXiv:2604. 26170v2 Announce Type: replace Abstract: Adapting large language models (LLMs) to a targeted task efficiently and effectively remains a fundamental challenge.
arXiv Machine Learning
Aug 28
Squeezing More from Limited Data with Recursive Transformers
The paper investigates how to effectively pre‑train language models when the data budget is limited but compute is plentiful. It shows that increasing model size only improves performance up to an optimal point, after which overfitting degrades generalization, and that this optimal size varies with both the data budget and downstream tasks. To overcome the inefficiencies of standard Transformers in this regime, the authors propose recursive Transformers that reuse a shared block across depth and employ factorized embeddings, achieving better results than standard models on 10M–100M word pre‑training budgets and competitive performance with BabyLM Challenge 2025 winners.
arXiv Machine Learning
Sep 11
Data Scarcity and Model Sparsity: Mixtures-of-Experts Overfit More to Repeated Data
The paper investigates how data repetition affects Mixture-of-Experts (MoE) language models compared to dense Transformers. Across models from 80 M to 1 B active parameters, MoEs degrade more quickly as data is repeated, with performance dropping significantly beyond 4× repetition and overtaking dense models only when strong regularization is applied. The study also identifies routing stabilization and expert specialization as key factors in MoE overfitting, and explores regularization techniques that can partially mitigate this issue.
arXiv Machine Learning
Aug 4
LangFIR: Discovering Sparse Language-Specific Features from Monolingual Data for Language Steering
arXiv:2604. 03532v2 Announce Type: replace-cross Abstract: Large language models (LLMs) show strong multilingual capabilities, yet reliably controlling the language of their outputs remains difficult.
arXiv AI
Jun 16
SPRI: SVD-Partitioned Residual Initialization for Data-Constrained MoE Upcycling
arXiv:2606. 16456v1 Announce Type: cross Abstract: Mixture-of-Experts (MoE) models enable efficient scaling, but training them from scratch remains prohibitively expensive.