← Back to all news
arXiv Machine Learning

Temperon: Full-Time SAM Quality at a Third Less Wall-Clock

The paper introduces Temperon, a training strategy that uses plain SGD for the first 43% of the epoch budget and then hands off to a SAM‑wrapped Muon refiner for the remaining training. On datasets such as CIFAR‑10/100, SVHN, and Tiny ImageNet, Temperon achieves the same or better accuracy as full‑time SAM while reaching key performance targets faster and at lower cost. Ablation studies show that the Muon refiner contributes the majority of the performance gain, while the initial SGD explorer and its restarts add negligible benefit.

Related stories

arXiv Computation and Language
3d ago

Tune the Baseline First: Most of Muon's Scaled Weight Decay Gain Is a Tuning Gap, but Its Small Real Advantage Grows With Model Size

The study re‑examines scaled weight decay (Muon‑SW) versus constant decoupled weight decay for language models, tuning learning‑rate and weight‑decay jointly across a range of model sizes. It finds that most of the reported performance gain is due to a tuning gap, with the tuned advantage shrinking to only 0.0026 BPB. However, the remaining advantage grows with model size, rising from 0.0036 to 0.0051 BPB over an eight‑fold increase in active parameters, and the optimal LR‑WD product shifts faster for Muon‑SW.

By Esther Xin
llms
More like this →
arXiv AI
Oct 16

When Does Muon Actually Win? A Paired Data-, Time-, and Tuning-Budget Benchmark

The paper evaluates Muon, a data‑efficient optimizer, against AdamW on a single A100 GPU across 17.7M‑ and 33.6M‑parameter causal language models and a 2.70M‑parameter vision transformer. Using a 23‑candidate development budget and five matched seeds, Muon consistently reduces validation‑loss area under the curve by 3.25–22.44% and lowers test loss in all 20 primary pairs, while AdamW outperforms Muon on language tasks by 3.38–6.96%. Muon’s full optimizer call consumes 56.1–69.6% of training time versus 6.1–9.1% for AdamW, and it shows higher expected regret under restricted candidate pools, indicating trade‑offs between data efficiency and computational cost for small models with reference kernels.

By Mushir Akhtar, Hazem N. Nounou, Majdi Mansouri, Mohamed N. Nounou
llmsbenchmarks
More like this →
arXiv Machine Learning
Sep 22

Muon Can Outperform Dedicated Continual Learning Methods

arXiv:2609.24678v1 Announce Type: new Abstract: Continual learning with Low-Rank Adapters (LoRA) typically mitigates forgetting by penalizing the overlap between a new update and the accumulated past...

By Sebastian George Sincari (Faculty of Mathematics and Computer Science, University of Bucharest, Bucharest, Romania), Bogdan Alexandru Gheorghe (Faculty of Mathematics and Computer Science, University of Bucharest, Bucharest, Romania), Antonio Barbalau (Bitdefender, Bucharest, Romania)
fine-tuningbenchmarks
More like this →
arXiv Machine Learning
Oct 19

The Ephemeral Cost of Switching Optimizers Across LLM Training Stages

The study investigates whether switching optimizers between pretraining and subsequent training stages of large language models (LLMs) incurs a performance penalty. By pretraining 1.2B Llama‑3 models with varying proportions of AdamW and Muon updates, and interpolating between the two during continued pretraining and supervised fine‑tuning (SFT), the authors find little evidence of a meaningful optimizer mismatch penalty. The results show that final performance is largely determined by the independent effects of the pretraining and post‑training optimizers, with downstream performance generally improving as the proportion of Muon updates during SFT increases, and that Muon remains competitive with AdamW even for task‑general SFT on larger models.

By Xinyu Zhao, Davis Wertheimer, Ashutosh Sharma, Rameswar Panda, Paul Pu Liang
llmsfine-tuningbenchmarks
More like this →
arXiv Machine Learning
Oct 28

End-to-End MXFP8 Reinforcement Learning on Blackwell: What Matching Buys, When It Pays Off, and How It Fails

The paper presents an end‑to‑end MXFP8 reinforcement learning implementation on Blackwell GPUs, evaluating its benefits and limitations with Qwen3-8B and Qwen3-30B-A3B models. It finds that matching FP8 precision between sampler and learner improves consistency, that speed gains appear only for large micro‑batches, and that several weight‑synchronization correctness conditions can fail silently. The study also introduces seven explicit checks to detect such failures during training.

By Wengee Zhang
llmsreinforcement-learningefficiency
More like this →
arXiv Computer Vision
Aug 24

When does fusing hand-crafted knowledge with learned representations pay? A cost-normalized benchmark of stacking, substitution, and interference

arXiv:2608.21098v1 Announce Type: new Abstract: Fusing prior knowledge with data-driven learning is attractive where data is scarce, yet no controlled account says when it helps, is redundant, or har...

By Ahmad AlMughrabi, Albert Clop, Benjamin Busam, Ricardo Marques, Petia Radeva
computer-visionbenchmarks
More like this →
arXiv Machine Learning
Oct 19

Peak-Memory Schedules You Can Trust: Exact Realization, Certified Optimality, and the Limits of Recomputation for Inference

The paper presents a method for reliably scheduling operator execution in real‑time inference on small devices, achieving exact peak‑memory predictions and certified optimality. By profiling the generated program and modeling its memory usage, the authors can permute statements to match the modeled peak, guaranteeing bitwise‑identical results on CPUs and characterizing GPU limitations. They also provide efficient lower‑bound traps and analyze selective recomputation, showing it is NP‑hard in general but can reduce peak memory on certain architectures.

By Ivan Fomin, Ivan Koryabkin, Boris Sheludko
More like this →