arXiv Machine Learning June 26, 2026 By Ionut-Vlad Modoranu, Philip Zmushko, Erik Schultheis, Mher Safaryan, Dan Alistarh

DASH: Faster Shampoo via Batched Block Preconditioning and Efficient Inverse-Root Solvers

Read the original on arXiv Machine Learning →

arXiv:2602. 02016v2 Announce Type: replace Abstract: Shampoo is one of the leading approximate second-order optimizers: a variant of it has won the MLCommons AlgoPerf competition, and it has been shown to produce models with lower activation outliers that are easier to compress.

Summary generated by The Flow from the publisher's feed. The full article lives at arXiv Machine Learning.

arXiv AI

Jun 2

FOAM: Frequency and Operator Error-Based Adaptive Damping Method for Reducing Staleness-Oriented Error for Shampoo

arXiv:2606. 02365v1 Announce Type: cross Abstract: Shampoo is attracting considerable attention for its superior performance on large-scale optimization benchmarks; yet it faces a significant practical bottleneck: the prohibitive computational overhead of matrix inversion.

By Kyunghun Nam, Sumyeong Ahn

benchmarks

arXiv Machine Learning

Jul 28

Low-Rank Dependence Decomposition via Accelerated Symmetric Non-negative Matrix Factorization

arXiv:2607. 24518v1 Announce Type: new Abstract: Symmetric non-negative matrix factorization (SymNMF) recovers latent group structure from a dependence matrix, but its dense, quadratic-memory objective has confined prior work to moderate sizes.

By Lavinia Ghita, Dhruv Desai, Jake Goldberg, Roman Yokunda Enzmann

benchmarks

arXiv AI

Jul 24

SOAP, Muon, and Beyond: Pushing LLM Pretraining Scales

arXiv:2607. 20548v1 Announce Type: cross Abstract: Higher-order optimizers such as Muon and SOAP offer faster convergence than AdamW, but their computational cost and numerical stability challenges have limited adoption at scale.

By Mikail Khona, Aditya Vavre, Boxiang Wang, Deyu Fu, Hao Wu, Mike Chrzanowski, Bryan Catanzaro, Dheevatsa Mudigere, Jeff Pool, Michael Lightstone, Mohammad Shoeybi, Mostofa Patwary, Nima Tajbakhsh, Tijmen Blankevoort

llms

arXiv Machine Learning

Jul 30

Between Gradient and Natural Gradient: A Continuum of LoRA Initializations

arXiv:2607. 26247v1 Announce Type: new Abstract: Low-rank adaptation (LoRA) fine-tunes large pretrained models at a fraction of the cost of full fine-tuning, but its performance depends strongly on how the adapters are initialized.

By Dianze Liu, Farshid Ghezelbash

llms fine-tuning

arXiv Machine Learning

Jun 3

Scalable Derivative Gaussian Processes via Exact Gradient Reduction

arXiv:2606. 02909v1 Announce Type: cross Abstract: Gradient observations can substantially improve Gaussian process (GP) surrogates, particularly in high-dimensional settings where function evaluations are expensive.

By Hyunseok Seung, Matthias Katzfuss

benchmarks

arXiv AI

Jul 29

At-the-Roofline Sparse Tensor Contractions on Vector Processors for Transformer Inference

arXiv:2607. 25504v1 Announce Type: cross Abstract: Fine-grained weight pruning and activation sparsification have emerged as effective approaches for reducing the compute and memory cost of inference for Transformer models.

By Bowen Wang, Chi Zhang, Diyou Shen, Renzo Andri, Navaneeth Kunhi Purayil, Luca Benini

llms efficiency

Related stories

FOAM: Frequency and Operator Error-Based Adaptive Damping Method for Reducing Staleness-Oriented Error for Shampoo

Low-Rank Dependence Decomposition via Accelerated Symmetric Non-negative Matrix Factorization

SOAP, Muon, and Beyond: Pushing LLM Pretraining Scales

Between Gradient and Natural Gradient: A Continuum of LoRA Initializations

Scalable Derivative Gaussian Processes via Exact Gradient Reduction

At-the-Roofline Sparse Tensor Contractions on Vector Processors for Transformer Inference