arXiv Machine Learning
Delayed Optimizer-State Transport Shapes Short-Horizon Training Decisions
Read the original on arXiv Machine Learning →The Flow has not summarised this story yet — read it at arXiv Machine Learning.
Related stories
arXiv Machine Learning
Sep 7
Optimizer Memory Schedules for Outscaling the Overtraining Axis
The paper studies how different optimizers perform as training duration (overtraining) increases, focusing on matrix‑preconditioned methods (Muon, SOAP) and a momentum‑scheduled method (ADANA) compared to AdamW. Across models ranging from 51M to 253M parameters and overtraining factors up to 256×, the authors find that optimal learning‑rate schedules, weight‑decay coefficients, and memory settings shift with horizon, and that ADANA consistently outperforms AdamW, especially with log‑time weight decay and momentum cooldown. Muon and SOAP maintain roughly constant token‑efficiency advantages, with SOAP potentially improving at the highest overtraining levels.
arXiv Machine Learning
1d ago
Error Correction in Trained TTT Layers
arXiv:2610.16268v2 Announce Type: replace Abstract: Test-time-training (TTT) layers update fast-weight memory while processing an input sequence. A gradient-based update, however, does not by itself...
arXiv Machine Learning
Jul 24
Adaptive Depth in Looped Transformers: Diagnosing Learned Halting Gates and Trajectory Readouts
arXiv:2607. 20519v1 Announce Type: new Abstract: Looped Transformers increase test-time computation by repeatedly applying a shared recurrent block.
arXiv Machine Learning
Jun 5
Double Preconditioning (DoPr): Optimization for Test-Time Performance, not Validation Loss
arXiv:2606. 06418v1 Announce Type: new Abstract: Many modern applications of deep learning involve training a neural network via a one-step prediction loss (e.
arXiv AI
Aug 21
Finite-Horizon Input-Output Dynamics of Minibatch Perturbations in AdamW
arXiv:2608. 19762v1 Announce Type: cross Abstract: A minibatch can influence training beyond the update at which it is observed because AdamW stores past gradient information in its optimizer states.
arXiv Machine Learning
Oct 5
The Conflict Between Logic and Memory: Training Conditions for Optimizer-Dependent Rule Acquisition
The paper investigates how different optimizers acquire distinct generalizing relations in single‑hidden‑layer ReLU networks. By manipulating nuisance‑connected weights and performing a strict‑parity sweep, the authors show that training conditions reshape both shared failures and relative optimizer advantages, with specific interventions improving accuracy on hard stages. The study also compares optimizer behavior across tasks and an external architecture (CIFAR‑10).