← Back to all news
arXiv Machine Learning By Aditya Somasundaram, Charles Guille-Escuret, Alexander Moreno, Zhengzhong Liu, Eric Xing

Toward a First-Principles Update Geometry for the Language-Model Head

Read the original on arXiv Machine Learning →

The paper proposes a new update geometry for the language‑model head by treating the head and softmax as a single module and using Hilbert’s projective distance to measure functional change. It replaces the spectral norm with the Euclidean row diameter, derives a tractable RowNorm update rule, and demonstrates that RowNorm substantially reduces step diameters and Hilbert perturbations while only slightly increasing validation loss.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

Related stories

arXiv AI
5d ago

MuonGS: Pretraining LLMs with Relaxed Weight-Norm Control

The paper introduces MuonGS, a variant of the Muon optimizer that employs Gram–Schmidt projection to remove the radial component of updates, thereby controlling weight norm growth without fixing it. By ensuring the weight norm grows at most proportionally to √{t}, MuonGS achieves a cumulative effective learning rate of ≥√{T} and an O(T^{-1/2}) stationarity guarantee for smooth scale‑invariant objectives. Experiments on 160M‑ and 400M‑parameter dense models and a 135M‑active/1B‑total MoE model show MuonGS attains lower validation perplexity than MuonH and an estimated token‑equivalent speedup of up to ~15% at matched validation loss.

By Athanasios Glentis, Dawei Li, Mingyi Hong
llms
More like this →
arXiv Machine Learning
Jun 3

MuLoCo: Muon is a practical inner optimizer for DiLoCo

arXiv:2505. 23725v3 Announce Type: replace Abstract: DiLoCo is a powerful framework for training large language models (LLMs), enabling larger optimal batch sizes and increased accelerator utilization under networking constraints.

By Benjamin Th\'erien, Xiaolong Huang, Aaron Defazio, Irina Rish, Eugene Belilovsky
llmsefficiency
More like this →