arXiv:2607. 27928v1 Announce Type: new Abstract: The performance of Large Language Models (LLMs) is fundamentally influenced by the distributional composition of multi-domain pre-training data.
arXiv:2603.16622v2 Announce Type: replace Abstract: Instead of directly distilling a language model, this study addresses the problem of aligning a base model with a target model in distribution by d...
arXiv:2607. 22769v1 Announce Type: cross Abstract: The training efficacy of large language models (LLMs) is fundamentally constrained by the quality and composition of training data.
arXiv:2602. 17554v3 Announce Type: replace Abstract: Training large-scale generative models is resource-intensive and relies heavily on heuristic dataset weighting.
arXiv:2607. 11052v1 Announce Type: new Abstract: Machine learning progress is often attributed to scaling model size and dataset volume, yet the composition of data can be just as consequential.
The paper introduces Power‑Law Entropy Search (PLES), a computational‑cost‑aware acquisition function that uses multi‑fidelity Bayesian optimization to efficiently estimate optimal hyperparameter scaling laws for large language model training. PLES focuses on reducing the overall uncertainty of scaling law estimates rather than optimizing a single objective, selecting configurations that maximize uncertainty reduction per unit computational cost. Experiments on synthetic benchmarks, surrogate models, and real LLM pre‑training runs show that PLES converges to accurate scaling laws using less than one‑tenth of the computational budget required by conventional grid search and other baselines.
arXiv:2607. 01104v1 Announce Type: cross Abstract: In Large Language Model (LLM) training, data mixing plays a pivotal role in determining model performance.
arXiv:2510. 01163v2 Announce Type: replace Abstract: The factors driving the performance of in-context learning (ICL) in large language models (LLMs) remain poorly understood despite ICL's surprising effectiveness, enabling models to adapt to new tasks from only a handful of examples.
arXiv:2608. 10867v1 Announce Type: new Abstract: Gradient-free post-training has emerged as a compelling alternative to gradient-based optimization for large language models (LLMs), but existing approaches remain costly.
arXiv:2606.22272v2 Announce Type: replace Abstract: Applying pre-trained language models to new domains through full fine-tuning is computationally expensive and prone to catastrophic forgetting. To...
arXiv:2607. 01686v1 Announce Type: new Abstract: Foundation models are routinely released to the public, yet the data recipes used to train them -- such as domain mixture weights that determine how different sources are sampled -- are rarely disclosed.
The paper introduces Bayesian Optimization (BO) techniques that incorporate rich auxiliary information—such as training curves, expert notes, images, and prior knowledge—using large language models (LLMs). Three new methods are proposed to integrate this auxiliary data into BO, and they are evaluated on hyperparameter optimization benchmarks and a real-world nuclear fusion task. The results show that these LLM-enhanced BO methods consistently outperform standard BO and existing LLM-based optimization approaches.
arXiv:2606. 08167v1 Announce Type: cross Abstract: Recent research has established empirical scaling laws to predict model performance on multi-domain data mixtures.