The paper proposes a framework to reduce the cost of constructing scaling laws for large foundation models by treating data collection as a Bayesian optimization problem. It shows that expanding the compute budget progressively and augmenting observed configurations with surrogate-fantasized evaluations can recover a broad experimental grid, enabling accurate scaling law fitting without training every configuration. This approach can achieve computational savings of up to 10–100× compared to a full dense grid.
By Abhash Kumar Jha, Diana Alexandra Onu\c{t}u, Neeratyoy Mallik, Swagatam Haldar, Sam Laing, Niccol\`o Ajroldi, Shiwei Liu, Joaquin Vanschoren, Aaron Klein
The paper introduces Power‑Law Entropy Search (PLES), a computational‑cost‑aware acquisition function that uses multi‑fidelity Bayesian optimization to efficiently estimate optimal hyperparameter scaling laws for large language model training. PLES focuses on reducing the overall uncertainty of scaling law estimates rather than optimizing a single objective, selecting configurations that maximize uncertainty reduction per unit computational cost. Experiments on synthetic benchmarks, surrogate models, and real LLM pre‑training runs show that PLES converges to accurate scaling laws using less than one‑tenth of the computational budget required by conventional grid search and other baselines.
By Zhiliang Chen, Sebastian Ament, David Eriksson, Maximilian Balandat, Eytan Bakshy, Jihao Andreas Lin
arXiv:2608. 03501v1 Announce Type: new Abstract: AI for Research (AI4Research) leverages AI to automate and improve scientific workflows.
By Zejun Liu, Jian Wu, Ru Peng, Yuliang Ji, Dongyuan Li, Renhe Jiang, Yue Zhang
The paper evaluates coreset selection methods by incorporating both selection and training time into a unified wall‑clock budget, using a standardized benchmark across four datasets and multiple selectors. Across numerous budget anchors, simple random or full‑data training consistently outperforms sophisticated selectors, and selection costs are dominated by a full‑dataset scan that cannot be amortized. The study also identifies when subset reuse can justify selection and reports several correctness fixes in a popular codebase.
By Yangze Liu, Zhongyi Han
arXiv:2608. 20061v1 Announce Type: cross Abstract: Mixture-of-Experts (MoE) architectures significantly expand model capacity without a proportional increase in computational cost.
By Nayeon Kim, Hojin Lee, Yunju Bak, Jaesun Park, Boseop Kim
arXiv:2606. 05186v1 Announce Type: new Abstract: Budget-constrained micro-pretraining often requires triaging many candidate recipes on a shared accelerator before larger search budgets are spent.
By Felipe Chavarro Polania
arXiv:2606.22826v2 Announce Type: replace
Abstract: Evaluating LLMs across many model variants---quantized, fine-tuned, or deployment-specific---requires running large benchmarks repeatedly, a proces...
By Devleena Das, Rajeev Patwari, Vikram Kumar Bukka, Nithin Kumar Guggilla, Elliott Delaye, Ashish Sirasao
arXiv:2608. 08189v1 Announce Type: new Abstract: LLM-driven program discovery relies on rapid evaluator feedback, but many scientific and engineering tasks require high-fidelity simulations, hardware execution, or physical experiments, making each evaluation expensive.
By Ximeng Liu, Qianlong Wang, Yingming Mao, Annan Li, Yatao Li, Shizhen Zhao, Jianmin Wu, Dawei Yin, Dou Shen
arXiv:2601. 12186v3 Announce Type: replace-cross Abstract: Multi-domain thinking verifiers trained via Reinforcement Learning with Verifiable Rewards (RLVR) are a cornerstone of modern post-training.
By Vatsal Venkatkrishna, Indraneil Paul, Iryna Gurevych
arXiv:2607. 09172v1 Announce Type: cross Abstract: Large Language Models are reshaping how software is developed and maintained.
By Nada Zine, Tristan Coignion, Vincenzo Stoico, Cl\'ement Quinton, Romain Rouvoy, Patricia Lago
arXiv:2602. 15327v2 Announce Type: replace-cross Abstract: Machine learning model performance improvements tend to arise from competition and application.
By Hanlin Zhang, Jikai Jin, Vasilis Syrgkanis, Sham Kakade
arXiv:2607. 25271v1 Announce Type: cross Abstract: Classical compute-optimal scaling laws assume an unbounded supply of fresh pretraining data, yet pretraining is increasingly entering a regime in which compute grows faster than the availability of high-quality data.
By Tian Qin, Kimia Hamidieh, David Alvarez-Melis