arXiv Statistics ML
Low-Rank Friction for Memory-Efficient Transformer Pretraining
Read the original on arXiv Statistics ML →The Flow has not summarised this story yet — read it at arXiv Statistics ML.
Related stories
arXiv Machine Learning
Aug 17
Friction-Augmented Drifting Models for Resource-Efficient Domain Translation
arXiv:2604. 18194v2 Announce Type: replace Abstract: Single-step generators promise high-fidelity synthesis at a fraction of the inference and training cost of ordinary differential equation (ODE)-based flow models, a central concern when compute is limited.
arXiv Machine Learning
Jul 16
Weight Feedback Computes the Jacobian Transpose Locally in Modern Deep Networks
arXiv:2607. 13380v1 Announce Type: new Abstract: Predictive Coding (PC) offers a biologically motivated alternative to backpropagation via local weight updates, yet routing error between layers still relies on an autograd Jacobian-transpose ($J^\top$) product - the last non-local operation in PC.
arXiv Machine Learning
Aug 18
On the Principles Behind Neural Network Optimizers
arXiv:2608. 16760v1 Announce Type: new Abstract: Reliable optimization is central to neural network (NN) training, yet Adam, the default optimizer for modern LLMs, rests on a fragile foundation.
arXiv Machine Learning
Sep 4
Activation-Keyed Momentum: An Anisotropic Momentum Update via the Delta Rule
The paper introduces Activation-Keyed Momentum (AK‑Momentum), a momentum update that uses the input activation of a linear layer as a key to apply a delta‑rule update, allowing each direction to decay at a rate proportional to its frequency of appearance. AK‑Momentum is proven to be a valid momentum, incorporates input‑side curvature correction without matrix inversion, and clears stale directions faster than traditional exponential moving average (EMA) under both fixed and drifting optima. It can replace the momentum buffer of any optimizer, scales with width under μP, adds only 22–25% extra compute, and demonstrates significant step‑count reductions in FineWeb‑Edu pretraining and other benchmarks. whyItMatters":"AK‑Momentum offers a principled, efficient way to adapt momentum decay to anisotropic training dynamics, improving convergence speed and stability across a range of models and datasets."
arXiv Machine Learning
Sep 15
A pullback-corrected scalar auxiliary variable optimizer with momentum and adaptive mobility
arXiv:2609.13569v1 Announce Type: cross Abstract: Objectives in scientific machine learning are often prescribed as a sum of several terms, such as the residual, boundary, initial, and data losses of...
arXiv Machine Learning
Aug 14
Training Non-Differentiable Networks via Optimal Transport
arXiv:2605. 01928v2 Announce Type: replace Abstract: We optimize losses that jump: spiking thresholds, quantized layers, and discrete routing put jumps in the forward pass, where backpropagation does not apply.