arXiv Machine Learning
Backward-State Policy Is Part of the Learning Algorithm
Read the original on arXiv Machine Learning →The Flow has not summarised this story yet — read it at arXiv Machine Learning.
Related stories
arXiv Machine Learning
Aug 21
DeltaMomentum: A Key-Value based Anisotropic Momentum Update via Delta Rule
arXiv:2608. 19491v1 Announce Type: new Abstract: Most modern optimizers form their momentum as an exponential moving average (EMA) of past gradients, forgetting every direction at one fixed rate.
arXiv Machine Learning
Sep 4
Activation-Keyed Momentum: An Anisotropic Momentum Update via the Delta Rule
The paper introduces Activation-Keyed Momentum (AK‑Momentum), a momentum update that uses the input activation of a linear layer as a key to apply a delta‑rule update, allowing each direction to decay at a rate proportional to its frequency of appearance. AK‑Momentum is proven to be a valid momentum, incorporates input‑side curvature correction without matrix inversion, and clears stale directions faster than traditional exponential moving average (EMA) under both fixed and drifting optima. It can replace the momentum buffer of any optimizer, scales with width under μP, adds only 22–25% extra compute, and demonstrates significant step‑count reductions in FineWeb‑Edu pretraining and other benchmarks. whyItMatters":"AK‑Momentum offers a principled, efficient way to adapt momentum decay to anisotropic training dynamics, improving convergence speed and stability across a range of models and datasets."
arXiv AI
Jul 17
Per-Token Fixed-Point Convergence in Depth-Recurrent Transformers
arXiv:2607. 14427v1 Announce Type: new Abstract: A depth-recurrent transformer applies a weight-tied core a variable number of times, and prior work has shown that training with a randomized recursion count yields one checkpoint usable across a range of inference depths.
arXiv Machine Learning
4d ago
Delta-Matching: Closing the Final Gap of Native 8-bit Training for LLMs
arXiv:2609.37852v1 Announce Type: new Abstract: Reliable FP8 attention remains a barrier to fully native 8-bit large language model training. We derive how forward-backward inconsistencies produce st...
arXiv Machine Learning
Jul 31
FORGE: Fused On-Register Gradient Elimination for Memory-Efficient LLM Training
arXiv:2606. 22932v2 Announce Type: replace Abstract: Reverse-mode differentiation computes every weight gradient, writes it to memory, and only then lets the optimizer read it back.
arXiv Machine Learning
Jul 14
Memory Savings at What Cost? A Study of Alternatives to Backpropagation
arXiv:2506. 21833v2 Announce Type: replace Abstract: Forward-mode automatic differentiation (FmAD) and zero-order (ZO) optimization are increasingly proposed as memory-efficient, backpropagation-free alternatives for large language model (LLM) fine-tuning, yet their benefits are typically evaluated only against standard backpropagation (BP), omitting memory-efficient variants such as activation checkpointing.