arXiv Machine Learning June 30, 2026 By John Sweeney

Optimizer Memory Makes Shuffle Order a First-Order Source of Fine-Tuning Noise

Read the original on arXiv Machine Learning →

arXiv:2606. 29554v1 Announce Type: new Abstract: Shuffle order can be a larger source of fine-tuning noise than a memoryless analysis predicts: fixed-clock optimizer memory makes local equal-multiset contrasts first order in the learning rate rather than second order, and the resulting order channel can be large enough for a single seed to flip a close A/B comparison.

Summary generated by The Flow from the publisher's feed. The full article lives at arXiv Machine Learning.

fine-tuning

arXiv Machine Learning

Jul 7

Variable Bit-width Quantization: Learning Per-Group Precision for "Bigger-but-Smaller" Language Models

arXiv:2607. 02893v1 Announce Type: new Abstract: Low-bit quantization shrinks language models but treats precision as a single global hyper-parameter: every weight uses the same bit-width.

By Hamish Ogilvy

llms efficiency

arXiv Machine Learning

Aug 4

On the Limits of Machine-Learned Ranking for Modern Microarchitectural Policies

arXiv:2608. 01041v1 Announce Type: cross Abstract: Machine-learning predictors estimate processor performance far faster than cycle-level simulation.

By Yanxin Zhang, Shayne Wadle, Yuxuan Xiong, Zheyu Fu, Trivikram Krishnamurthy, Karu Sankaralingam

arXiv Machine Learning

Jul 21

PsiLogic: Chaos-Aware Active Cancellation for Adam with a Fair Cross-Domain Benchmark

arXiv:2607. 16268v1 Announce Type: new Abstract: Adaptive optimizers such as Adam and AdamW apply the same update rule regardless of whether training is in a chaotic early phase or near convergence.

By Ali Sultonov

llms diffusion nlp benchmarks

arXiv Machine Learning

Jul 17

Adaptive Runge-Kutta Step Control Buys Training Loss, Not Generalization: An Honest Compute-Matched Study of RK-Adam Optimizers

arXiv:2607. 14516v1 Announce Type: new Abstract: Interpreting optimizers as gradient-flow discretizations has motivated applying higher-order Runge-Kutta (RK) integrators to neural networks.

By Akhilesh Gogikar

arXiv AI

Aug 6

MESH: Memory-Efficient Sinkhorn Optimization for Mixture-of-Experts Training

arXiv:2608. 04407v1 Announce Type: cross Abstract: Memory-efficient matrix optimizers such as Sinkhorn gradient descent remove most AdamW optimizer state for dense Transformer matrices, but direct application to Mixture-of-Experts (MoE) training is unreliable.

By Masato Fujitake

llms

arXiv Machine Learning

Jul 14

Memory Savings at What Cost? A Study of Alternatives to Backpropagation

arXiv:2506. 21833v2 Announce Type: replace Abstract: Forward-mode automatic differentiation (FmAD) and zero-order (ZO) optimization are increasingly proposed as memory-efficient, backpropagation-free alternatives for large language model (LLM) fine-tuning, yet their benefits are typically evaluated only against standard backpropagation (BP), omitting memory-efficient variants such as activation checkpointing.

By Kunjal Panchal, Sunav Choudhary, Yuriy Brun, Hui Guan

llms fine-tuning multimodal benchmarks

Related stories

Variable Bit-width Quantization: Learning Per-Group Precision for "Bigger-but-Smaller" Language Models

On the Limits of Machine-Learned Ranking for Modern Microarchitectural Policies

PsiLogic: Chaos-Aware Active Cancellation for Adam with a Fair Cross-Domain Benchmark

Adaptive Runge-Kutta Step Control Buys Training Loss, Not Generalization: An Honest Compute-Matched Study of RK-Adam Optimizers

MESH: Memory-Efficient Sinkhorn Optimization for Mixture-of-Experts Training

Memory Savings at What Cost? A Study of Alternatives to Backpropagation