arXiv Machine Learning
Delayed Optimizer-State Transport Shapes Short-Horizon Training Decisions
Read the original on arXiv Machine Learning →The Flow has not summarised this story yet — read it at arXiv Machine Learning.
Related stories
arXiv Machine Learning
Jul 24
Adaptive Depth in Looped Transformers: Diagnosing Learned Halting Gates and Trajectory Readouts
arXiv:2607. 20519v1 Announce Type: new Abstract: Looped Transformers increase test-time computation by repeatedly applying a shared recurrent block.
arXiv AI
Aug 21
Finite-Horizon Input-Output Dynamics of Minibatch Perturbations in AdamW
arXiv:2608. 19762v1 Announce Type: cross Abstract: A minibatch can influence training beyond the update at which it is observed because AdamW stores past gradient information in its optimizer states.
arXiv Machine Learning
Aug 27
Spectral Allocation: Why Muon Outperforms Adam, and How to Improve Muon
The paper investigates why the orthogonal optimiser Muon outperforms Adam in large language model pretraining by analysing the spectral properties of Transformer loss landscapes. It finds that Muon’s momentum buffers exhibit an anisotropic spectral profile with a volatile head and a tolerant bulk, enabling larger effective step sizes. Building on this insight, the authors propose Spectral‑Aware Muon (SAMuon) and a lightweight variant, which adjust the bulk scaling while keeping the head unchanged, achieving 13–24 % fewer training tokens than Muon without extra FLOPs.
arXiv AI
Aug 6
MESH: Memory-Efficient Sinkhorn Optimization for Mixture-of-Experts Training
arXiv:2608. 04407v1 Announce Type: cross Abstract: Memory-efficient matrix optimizers such as Sinkhorn gradient descent remove most AdamW optimizer state for dense Transformer matrices, but direct application to Mixture-of-Experts (MoE) training is unreliable.
arXiv AI
Jul 21
First-Order Predictable but Pairwise Fragile: Local Task Adaptation in Trained Transformers
arXiv:2607. 16821v1 Announce Type: cross Abstract: Task arithmetic, sequential fine-tuning, activation steering, and first-order random search all operate through relatively small perturbations around an already trained checkpoint, and they rely on different local approximations: individual perturbations should be first-order predictable, task updates should compose with controlled interference, useful tangent structure should be stable and possible to estimate, and weight edits should have counterparts in representation space.
arXiv Machine Learning
Aug 24
Stored in Optimizer State, Valued by Later Training: A Causal Account of Subliminal Trait Transfer
arXiv:2608.20442v1 Announce Type: new Abstract: Subliminal trait transfer allows a student model to acquire behavioral dispositions from teacher-generated data in which the trait is not semantically...