arXiv:2607. 13380v1 Announce Type: new Abstract: Predictive Coding (PC) offers a biologically motivated alternative to backpropagation via local weight updates, yet routing error between layers still relies on an autograd Jacobian-transpose ($J^\top$) product - the last non-local operation in PC.
By Junlong Shen, Xingyu Li
arXiv:2608. 19491v1 Announce Type: new Abstract: Most modern optimizers form their momentum as an exponential moving average (EMA) of past gradients, forgetting every direction at one fixed rate.
By Euijin Hong, Guannan Qu
The paper introduces Activation-Keyed Momentum (AK‑Momentum), a momentum update that uses the input activation of a linear layer as a key to apply a delta‑rule update, allowing each direction to decay at a rate proportional to its frequency of appearance. AK‑Momentum is proven to be a valid momentum, incorporates input‑side curvature correction without matrix inversion, and clears stale directions faster than traditional exponential moving average (EMA) under both fixed and drifting optima. It can replace the momentum buffer of any optimizer, scales with width under μP, adds only 22–25% extra compute, and demonstrates significant step‑count reductions in FineWeb‑Edu pretraining and other benchmarks.
whyItMatters":"AK‑Momentum offers a principled, efficient way to adapt momentum decay to anisotropic training dynamics, improving convergence speed and stability across a range of models and datasets."
By Euijin Hong, Guannan Qu
arXiv:2605.07815v2 Announce Type: replace
Abstract: Muon fixes the \emph{direction} of every matrix-valued update at the polar factor of its momentum, while each layer's step \emph{magnitude} is addr...
By Yuxuan Lou, Yang You
arXiv:2608. 02991v1 Announce Type: new Abstract: Matrix spectral optimizers reshape weight-update spectra but usually delegate vector-valued biases to a separate optimizer.
By Gongyue Zhang, Honghai Liu
arXiv:2608.08322v2 Announce Type: replace-cross
Abstract: Adaptive loss-balancing schemes for physics-informed neural networks rest on a premise that every residual should be driven to zero. For leve...
By Muhammad Akbar Khan