The paper introduces a new closed‑form scaling law that extends Chinchilla’s original formula to handle data‑constrained regimes. It decomposes loss into undercapacity, undertraining, and overfitting components, saturating between an irreducible loss and an uninformed baseline. The authors validate the model on diverse architectures and domains, achieving state‑of‑the‑art RMSE across multiple LLM scaling‑law grids and enabling cost‑aware training allocations.
By Christopher M. Bryant, Hao Liu
arXiv:2606. 29158v1 Announce Type: cross Abstract: Learning-rate transfer can reduce the cost of training large language models: instead of sweeping learning rates at target scale, practitioners extrapolate from smaller runs.
By Zaiwen Yang, Huaqing Zhang, Jing Xu, Jingzhao Zhang
arXiv:2512. 22088v3 Announce Type: replace-cross Abstract: The scaling law, a cornerstone of Large Language Model (LLM) development, predicts improvements in model performance with increasing computational resources.
By Chiwun Yang
arXiv:2607. 22757v1 Announce Type: cross Abstract: We introduce Graded Large Language Models (GLLMs), an algebraic framework that equips the representation space of a transformer with a grading and propagates the induced weighted scalar action through embeddings, self-attention, and the training objective.
By T. Shaska
The paper investigates online linear regression with sparse comparators, focusing on feature priming techniques that reweight features using past data. It establishes sparse‑regret lower bounds that invalidate sparse‑logarithmic guarantees for univariate, Pearson, and multivariate priming rules under a past‑only Moore–Penrose protocol, showing ≥Ω(min{T,√d}) clipped regret for unit‑power rules and linear regret for powered rules in high dimensions. The authors also provide tight rank upper bounds for certain priming schemes and present algebraic constructions yielding Ω(min{T,d^{1/4}}) regret for unit‑power multivariate priming, while noting that the exact multivariate frontier remains open.
By Huibo Xu, Shi Fu, Qixin Zhang, Dacheng Tao
arXiv:2608. 08826v1 Announce Type: new Abstract: Adaptive procedures must work without nuisance information an oracle may use, such as a gradient scale or smoothness index, and robust procedures may have to answer queries whose coordinate and inspection time are chosen only after the data are seen.
By Ibne Farabi Shihab, Adria Binte Habib
arXiv:2607. 26577v1 Announce Type: new Abstract: Adaptive conformal inference (ACI) of Gibbs and Cand{\`e}s and its variants are the standard approach to online conformal prediction under distribution shift, but they suffer from three fundamental limitations.
By Rahul Vaze
arXiv:2602. 07488v3 Announce Type: replace-cross Abstract: Despite the fact that experimental neural scaling laws have substantially guided empirical progress in large-scale machine learning, no existing theory can quantitatively predict the exponents of these important laws for any modern LLM trained on any natural language dataset.
By Francesco Cagnetta, Allan Ravent\'os, Surya Ganguli, Matthieu Wyart
arXiv:2607. 21866v1 Announce Type: new Abstract: Prior classical-ML learning-curve work fits power laws to tree, linear, and kernel models on tabular data, but at small scale: typically one curve, one team, a handful of cells.
By Kaihua Ding
arXiv:2512. 19905v3 Announce Type: replace-cross Abstract: Recent developments in large language models have shown advantages in reallocating a notable share of computational resource from training time to inference time.
By Indranil Halder, Cengiz Pehlevan
arXiv:2609. 22783v1 Announce Type: new Abstract: We study differentially private covariance estimation in operator norm for mean-zero sub-Gaussian distributions with unknown covariance support and at most $k$ nonzero entries per row.
By Zihan Zhang
arXiv:2607. 25271v1 Announce Type: cross Abstract: Classical compute-optimal scaling laws assume an unbounded supply of fresh pretraining data, yet pretraining is increasingly entering a regime in which compute grows faster than the availability of high-quality data.
By Tian Qin, Kimia Hamidieh, David Alvarez-Melis