arXiv Machine Learning

Coupled Scaling: A Representational Accessibility Framework for Neural Scaling Laws

The paper introduces Coupled Scaling, a framework that links neural scaling laws to the relationship between task structure and the geometry that an architecture‑optimization system can access. It shows that finite‑budget scaling depends on how well the system’s representational support aligns with the task’s energy distribution, deriving residual exponents that vary with architectural coverage and tail decay. The authors propose tests to verify whether static task‑relevant geometry tracks loss and whether multiscale geometry follows coupling‑specific exponent ordering, suggesting a factorial audit of emergence trajectories to isolate geometry from scaling fits.

arXiv Machine Learning

Sep 23

Practical Scaling Laws: Converting Compute into Performance in a Data-Constrained World

The paper introduces a new closed‑form scaling law that extends Chinchilla’s original formula to handle data‑constrained regimes. It decomposes loss into undercapacity, undertraining, and overfitting components, saturating between an irreducible loss and an uninformed baseline. The authors validate the model on diverse architectures and domains, achieving state‑of‑the‑art RMSE across multiple LLM scaling‑law grids and enabling cost‑aware training allocations.

By Christopher M. Bryant, Hao Liu

llms benchmarks

arXiv Machine Learning

1d ago

Optimizer-dependent training dynamics converge to the same one-third optimal data scaling

arXiv:2609.37745v1 Announce Type: new Abstract: Neural scaling, in which loss falls as a power law with training, is central to large language models, and one recent proposal is that a $1/3$ exponent...

By Hyunseok Lee, Mihir Basil, Yizhou Liu, Jeff Gore

llms safety

arXiv AI

Jul 21

First-Order Predictable but Pairwise Fragile: Local Task Adaptation in Trained Transformers

arXiv:2607. 16821v1 Announce Type: cross Abstract: Task arithmetic, sequential fine-tuning, activation steering, and first-order random search all operate through relatively small perturbations around an already trained checkpoint, and they rely on different local approximations: individual perturbations should be first-order predictable, task updates should compose with controlled interference, useful tangent structure should be stable and possible to estimate, and weight edits should have counterparts in representation space.

By Irina Piontkovskaia, Sergey Nikolenko

llms fine-tuning

arXiv AI

Aug 3

The Capability Convergence Hypothesis: Capability from Access Structure, Not Scale

arXiv:2607. 14144v2 Announce Type: replace Abstract: The Platonic Representation Hypothesis (PRH) holds that as models scale, representations of heterogeneous networks converge toward a shared model of reality.

By Wenhui Chen, Jianlin Chen, Ziyao Lin, Chi Man Vong

arXiv Machine Learning

Aug 12

Power law graph attention: exact generalization of scaled dot-product attention, empirical collapse at inference

arXiv:2608. 10288v1 Announce Type: new Abstract: The Large Language Model from Power Law Decoder Representations (PLDR-LLM) and its attention, Power Law Graph Attention (PLGA), replace the fixed bilinear form of scaled dot-product attention (SDPA) with a learned, input-generated bilinear operator $G_{LM}$, built from a positive tensor $A_{LM}$ by elementwise power laws.

By Burc Gokden

llms

arXiv AI

Jul 24

When Does Recurrence Become an Algorithm? Convergence Selection in Weight-Tied Looped Transformers

arXiv:2607. 20594v1 Announce Type: cross Abstract: When does a weight-tied looped transformer -- one block applied T times -- implement an actual algorithm?

By Tong Zhang, Junhao Hu, Yun Peng, Tao Xie

llms benchmarks

arXiv Machine Learning

Sep 1

OrScale: Orthogonalised Optimization with Layer-Wise Trust-Ratio Scaling

arXiv:2605.07815v2 Announce Type: replace Abstract: Muon fixes the \emph{direction} of every matrix-valued update at the polar factor of its momentum, while each layer's step \emph{magnitude} is addr...

By Yuxuan Lou, Yang You

benchmarks

arXiv AI

Jul 17

Capability from Access Structure, Not Scale: Lower Bounds and Pre-Registered Tests for Hybrid Sequence Models

arXiv:2607. 14144v1 Announce Type: new Abstract: The Platonic Representation Hypothesis (PRH) holds that as models scale, representations of heterogeneous networks converge toward a shared model of reality.

By Wenhui Chen, Jianlin Chen, Ziyao Lin, Chi Man Vong

arXiv AI

Jun 9

Item Response Scaling Laws: A Measurement Theory Approach for Efficient and Generalizable Neural Scaling Estimation

arXiv:2606. 07616v1 Announce Type: cross Abstract: Scaling laws provide a fundamental framework for understanding the performance of Language Models (LMs), yet deriving them requires prohibitively expensive evaluations across thousands of checkpoints or millions of inference samples.

By Sang Truong, Yuheng Tu, Rylan Schaeffer, Sanmi Koyejo

llms benchmarks

arXiv AI

Jul 23

An Isotropy-Preserving Spectral Cap for Muon: Theory and Three Case Studies

arXiv:2607. 19771v1 Announce Type: cross Abstract: Muon and related matrix-sign optimizers are increasingly used to pre-train large language models, but their effect on the internal geometry of individual weight matrices is not well understood.

By Jiachun Li

llms

arXiv Machine Learning

Aug 14

Neural Quadratic Forms: A Unified Minimal Model for Sudden Learning and Scaling Laws

arXiv:2608. 13335v1 Announce Type: new Abstract: Neural networks trained by gradient descent on a smooth cost function can nevertheless learn in steps: the cost holds on long plateaus and then drops abruptly.

By Liu Ziyin, Yizhou Xu, Tomaso Poggio, Isaac Chuang

arXiv AI

Aug 21

Let's Scale Step by Step: Compute-Efficient Hyperparameter Transfer for Large-Scale Mixture-of-Experts

arXiv:2608. 20061v1 Announce Type: cross Abstract: Mixture-of-Experts (MoE) architectures significantly expand model capacity without a proportional increase in computational cost.

By Nayeon Kim, Hojin Lee, Yunju Bak, Jaesun Park, Boseop Kim

benchmarks

Related stories