← Back to all news
arXiv Computation and Language By Hubert Baniecki, Przemyslaw Biecek, Martin Pawelczyk

LinMix: Data Mixing as Linear Domain Attribution

Read the original on arXiv Computation and Language →

LinMix proposes a hyperparameter‑free vertex sampling strategy for data mixing in language model pretraining, treating domain attribution as a linear regression problem. By sampling mixes at the vertices of a capped simplex, it decorrelates domain effects and yields an interpretable linear model whose coefficients directly reflect each domain’s contribution. Experiments on 1B‑parameter OLMo2 and 4B Qwen3.5 models demonstrate that LinMix outperforms traditional Dirichlet‑based mixing, achieving significant gains on reasoning, code, and math benchmarks and improving overall accuracy by 11% on average over the best nonlinear baseline.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computation and Language.

Related stories

arXiv AI
Aug 26

Data Mixing as Mixture Experiment: Response Surface Methodology and Optimal Design for Large Language Model Pretraining

The article proposes treating large language model (LLM) data mixing as a classical mixture experiment, where data domains are components, token shares are proportions, and proxy-training runs serve as design points. Using sparse second‑order Scheffé response‑surface models, the authors construct model‑robust Σ‑optimal designs that efficiently identify optimal data mixtures and reveal strong interaction effects, especially between weak domains and web‑derived text. Empirical results on RegMix show that these designs recover mixture rankings while reducing proxy runs by about 25%, demonstrating that data mixing can be optimized through experimental design rather than solely prediction.

By Yicheng Mao, Hongru Du
llms
More like this →
arXiv AI
Oct 28

An Embarrassingly Simple Data Selection Strategy for Million-Scale Multi-Modal Instruction-Tuning

The paper introduces SSR, a simple yet effective data selection strategy for million‑scale multi‑modal instruction‑tuning. SSR uses interaction styles as a proxy for structural diversity and decomposes data quality into capability‑specific scores, forming a rich profile for each sample. By casting selection as submodular coverage maximization, SSR consistently outperforms random sampling across two architecture stacks, multiple data pools, and 14 benchmarks, achieving 99.1% of full‑data performance with only 30% of a 2.6M pool.

By Mengyao Lyu, Yan Li, Huasong Zhong, Hui Chen, Jungong Han, Guiguang Ding, Zhenheng Yang
llmsragfine-tuningmultimodalbenchmarkssafetydata-engineering
More like this →
arXiv Machine Learning
Oct 16

INCONVENIENT: A Benchmark to Assess LLM Generalization Capabilities from Convenience Samples

INCONVENIENT is a new benchmark that evaluates how machine‑learning models, especially large language models, generalize when trained on convenience samples that bias feature–target relationships. Unlike prior distribution‑shift tests that rely on natural domain splits, INCONVENIENT explicitly perturbs these relationships while keeping the deployment population unchanged, allowing controlled study of shift severity and direction. The benchmark covers 200 tabular classification and regression datasets and produces about 15,500 shifted training distributions, revealing significant performance degradation.

By Sara Rezaeimanesh, Chad Klochko, Kundan Thind, Mohammad Ghassemi
llmsbenchmarkssafety
More like this →