← Back to all news
arXiv Machine Learning By Stamatis Mastromichalakis

Temperon: Full-Time SAM Quality at a Third Less Wall-Clock

Read the original on arXiv Machine Learning →

The paper introduces Temperon, a training strategy that uses plain SGD for the first 43% of the epoch budget and then hands off to a SAM‑wrapped Muon refiner for the remaining training. On datasets such as CIFAR‑10/100, SVHN, and Tiny ImageNet, Temperon achieves the same or better accuracy as full‑time SAM while reaching key performance targets faster and at lower cost. Ablation studies show that the Muon refiner contributes the majority of the performance gain, while the initial SGD explorer and its restarts add negligible benefit.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

Related stories

arXiv Computation and Language
3d ago

Tune the Baseline First: Most of Muon's Scaled Weight Decay Gain Is a Tuning Gap, but Its Small Real Advantage Grows With Model Size

The study re‑examines scaled weight decay (Muon‑SW) versus constant decoupled weight decay for language models, tuning learning‑rate and weight‑decay jointly across a range of model sizes. It finds that most of the reported performance gain is due to a tuning gap, with the tuned advantage shrinking to only 0.0026 BPB. However, the remaining advantage grows with model size, rising from 0.0036 to 0.0051 BPB over an eight‑fold increase in active parameters, and the optimal LR‑WD product shifts faster for Muon‑SW.

By Esther Xin
llms
More like this →
arXiv AI
Oct 16

When Does Muon Actually Win? A Paired Data-, Time-, and Tuning-Budget Benchmark

The paper evaluates Muon, a data‑efficient optimizer, against AdamW on a single A100 GPU across 17.7M‑ and 33.6M‑parameter causal language models and a 2.70M‑parameter vision transformer. Using a 23‑candidate development budget and five matched seeds, Muon consistently reduces validation‑loss area under the curve by 3.25–22.44% and lowers test loss in all 20 primary pairs, while AdamW outperforms Muon on language tasks by 3.38–6.96%. Muon’s full optimizer call consumes 56.1–69.6% of training time versus 6.1–9.1% for AdamW, and it shows higher expected regret under restricted candidate pools, indicating trade‑offs between data efficiency and computational cost for small models with reference kernels.

By Mushir Akhtar, Hazem N. Nounou, Majdi Mansouri, Mohamed N. Nounou
llmsbenchmarks
More like this →
arXiv Machine Learning
Sep 22

Muon Can Outperform Dedicated Continual Learning Methods

arXiv:2609.24678v1 Announce Type: new Abstract: Continual learning with Low-Rank Adapters (LoRA) typically mitigates forgetting by penalizing the overlap between a new update and the accumulated past...

By Sebastian George Sincari (Faculty of Mathematics and Computer Science, University of Bucharest, Bucharest, Romania), Bogdan Alexandru Gheorghe (Faculty of Mathematics and Computer Science, University of Bucharest, Bucharest, Romania), Antonio Barbalau (Bitdefender, Bucharest, Romania)
fine-tuningbenchmarks
More like this →