arXiv Machine Learning July 31, 2026 By Xiang Yuan, Kaiqing Lei, Zhenyu Jin, Jun Shu, Deyu Meng, Zongben Xu

Harnessing the Potential of Optimizing Data Mixtures via Bayesian Domain Reweighting

Read the original on arXiv Machine Learning →

arXiv:2607. 27928v1 Announce Type: new Abstract: The performance of Large Language Models (LLMs) is fundamentally influenced by the distributional composition of multi-domain pre-training data.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

arXiv Computation and Language

Sep 1

Domain Mixture Design via Log-Likelihood Differences for Aligning Language Models with a Target Model

arXiv:2603.16622v2 Announce Type: replace Abstract: Instead of directly distilling a language model, this study addresses the problem of aligning a base model with a target model in distribution by d...

By Ryo Kishino, Riku Shiomi, Hiroaki Yamagiwa, Momose Oyama, Hidetoshi Shimodaira

llms efficiency safety

arXiv AI

Jul 28

DomainPilot: Domain-Level Loss-Guided Two-Stage Data Mixture Optimization for Efficient Language Model Fine-Tuning

arXiv:2607. 22769v1 Announce Type: cross Abstract: The training efficacy of large language models (LLMs) is fundamentally constrained by the quality and composition of training data.

By He Zhang

llms fine-tuning benchmarks data-engineering

arXiv Machine Learning

Aug 13

A Theoretical Framework for Modular Learning of Robust Generative Models

arXiv:2602. 17554v3 Announce Type: replace Abstract: Training large-scale generative models is resource-intensive and relies heavily on heuristic dataset weighting.

By Corinna Cortes, Mehryar Mohri, Yutao Zhong

llms efficiency

arXiv Machine Learning

Jul 14

Domain-Aware Scaling Laws Uncover Data Synergy

arXiv:2607. 11052v1 Announce Type: new Abstract: Machine learning progress is often attributed to scaling model size and dataset volume, yet the composition of data can be just as consequential.

By Kimia Hamidieh, Lester Mackey, David Alvarez-Melis

llms benchmarks

arXiv AI

Sep 2

Efficiently Estimating Optimal Hyperparameter Scaling Laws through Power-Law Entropy Search

The paper introduces Power‑Law Entropy Search (PLES), a computational‑cost‑aware acquisition function that uses multi‑fidelity Bayesian optimization to efficiently estimate optimal hyperparameter scaling laws for large language model training. PLES focuses on reducing the overall uncertainty of scaling law estimates rather than optimizing a single objective, selecting configurations that maximize uncertainty reduction per unit computational cost. Experiments on synthetic benchmarks, surrogate models, and real LLM pre‑training runs show that PLES converges to accurate scaling laws using less than one‑tenth of the computational budget required by conventional grid search and other baselines.

By Zhiliang Chen, Sebastian Ament, David Eriksson, Maximilian Balandat, Eytan Bakshy, Jihao Andreas Lin

llms benchmarks

arXiv AI

Jul 2

CausalMix: Data Mixture as Causal Inference for Language Model Training

arXiv:2607. 01104v1 Announce Type: cross Abstract: In Large Language Model (LLM) training, data mixing plays a pivotal role in determining model performance.

By Zinan Tang, Yukun Zhang, Shaomian Zheng, Zhuoshi Pan, Qizhi Pei, Dingnan Jin, Jun Zhou, Yujun Wang, Biqing Huang

llms

One email a morning, machine-written

Related stories

Domain Mixture Design via Log-Likelihood Differences for Aligning Language Models with a Target Model

DomainPilot: Domain-Level Loss-Guided Two-Stage Data Mixture Optimization for Efficient Language Model Fine-Tuning

A Theoretical Framework for Modular Learning of Robust Generative Models

Domain-Aware Scaling Laws Uncover Data Synergy

Efficiently Estimating Optimal Hyperparameter Scaling Laws through Power-Law Entropy Search

CausalMix: Data Mixture as Causal Inference for Language Model Training