The paper introduces the Drift Contract, a spectral update geometry for local learning that improves depth robustness and hyperparameter stability. By applying momentum orthogonalization with spectral step scaling to per‑layer updates, the authors achieve consistent performance across a wide range of widths and depths on CIFAR‑10 MLPs, outperforming local Adam and providing a per‑layer, input‑conditioned drift bound. The study also shows that the spectral geometry itself, rather than step‑size rules, drives the observed depth robustness, while a negative result indicates that the stability benefit is limited to non‑normalized layers.
By Fabien Polly
arXiv:2607. 23711v1 Announce Type: new Abstract: LoRA fine-tuning can create intruder dimensions: new leading singular vectors of the updated weight matrix $W+BA$ that are nearly orthogonal to all pretrained singular vectors and that drive catastrophic forgetting.
By Peng Xie
arXiv:2608. 04084v1 Announce Type: new Abstract: Mixture-of-experts (MoE) networks pursue specialization through learned routers, gates, and load-balancing losses, yet at matched total-parameter budgets learned routers can underperform equal-weight No-Routing baselines.
By Boyao Wang, Zhihan Lei
The paper investigates how the usefulness of training samples, as determined by coreset selection, depends on the learner rather than just the data. Experiments on ImageNet-100 and ImageNet-1k show that changing model width, input grid, stride, and architecture (e.g., ResNet vs. ViT) shifts the crossover point where different selection criteria (easy-first vs. geometric coverage) become optimal. These findings demonstrate that the relative value of a fixed subset of samples varies with the target learner’s capacity and structure, and that selection strategies must be tuned to the specific model they will train.
By Yangze Liu, Xiao-Long Yin, Zhongyi Han
The paper revisits the interpretation of depth profiles in Diffusion Transformer backbones, showing that the commonly observed inverted‑U curve reflects the influence of the prediction target (e.g., ε, u, x0) rather than a simple shift from coarse to fine semantic granularity. It argues that the semantically meaningful signal is merely an intermediate by‑product of predicting the target, and that different training setups—particularly the choice of prediction target—predictably alter the curve’s shape. The authors demonstrate how this new understanding can guide the optimal placement of REPA modules and provide code for further exploration.
By Benyuan Meng, Longtao Huang, Yang Liu, Liang Lin
arXiv:2605. 17109v3 Announce Type: replace-cross Abstract: In recent years, Muon has emerged as the dominant method for training large language models, and transformers more broadly.
By Fangzhou Wu, Rikhav Shah, Sandeep Silwal, Qiuyi Zhang
The paper studies generative flows that sample from an unnormalized target by training a flow to satisfy a balance condition. It shows that the training loss can certify sampler accuracy, can be driven to zero, and quantifies how fast gradient descent converges, with explicit bounds depending on the backward policy and graph structure. The authors provide theoretical guarantees, test them on finite graphs, and certify each theorem using a Lean 4 development.
By Leo Brunswic
arXiv:2607. 23777v1 Announce Type: cross Abstract: The discovery of scaling laws has motivated training neural networks on ever increasing quantities of data.
By Anuj Apte
arXiv:2606. 12883v1 Announce Type: new Abstract: In Low-Rank Adaptation (LoRA), the scaling factor $\alpha$ is often treated as a mere complement to the learning rate, yet its role in optimization remains poorly understood.
By Zicheng Zhang, Haoran Li, Jiaxing Wang, Guoqiang Gong, Anqi Li, Yudong Hu, Ting Xiong, Yurong Gao, Junxing Hu, Zhida Jiang, Yifeng Zhang, Pengzhang Liu, Qixia Jiang
The paper introduces a new closed‑form scaling law that extends Chinchilla’s original formula to handle data‑constrained regimes. It decomposes loss into undercapacity, undertraining, and overfitting components, saturating between an irreducible loss and an uninformed baseline. The authors validate the model on diverse architectures and domains, achieving state‑of‑the‑art RMSE across multiple LLM scaling‑law grids and enabling cost‑aware training allocations.
By Christopher M. Bryant, Hao Liu
arXiv:2608. 05136v1 Announce Type: new Abstract: Gradient descent on a factored model $W = UV^\top$ is implicitly biased toward low-rank solutions, while Adam, starting from the same small initialization, is not.
By Devender Singh
arXiv:2610. 11730v1 Announce Type: new Abstract: Standard weight decay treats each weight matrix as a vector and ignores its spectral structure.
By Dmitrii Andriianov, Andrey Veprikov, Aleksandr Beznosikov