arXiv AI
Depth and Scale in the Sub-150M Regime: JugnuLM-53M vs JugnuLM-110M
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv Machine Learning
Sep 3
Train What You Deploy: Closing the MLP Reachability Gap in Low-Rank Clone Distillation
The paper introduces a new approach to low‑rank clone distillation that ensures the student model’s training targets exactly the weights it will use at inference. By redefining the training objective to cover the full deployed matrix—without changing the model’s shape, parameter count, or FLOPs—the authors recover previously unreachable linear degrees of freedom. This results in significant performance gains across multiple teacher models, achieving comparable or superior accuracy with fewer tokens and parameters.
arXiv Machine Learning
Aug 5
Wiring Beats Blending: What Transfers Between Transformer Sizes -- and What Doesn't
arXiv:2608. 02829v1 Announce Type: new Abstract: Model families train every size from scratch.
arXiv Machine Learning
Aug 12
CurveFP: Rational-Radix Logarithmic Datatypes with Closed Products for Language Models
arXiv:2608. 10010v1 Announce Type: new Abstract: Low-precision datatypes reduce language-model cost, but most formats optimize scalar fidelity while leaving the arithmetic induced by their products unchanged.
arXiv Machine Learning
Sep 7
Optimizer Memory Schedules for Outscaling the Overtraining Axis
The paper studies how different optimizers perform as training duration (overtraining) increases, focusing on matrix‑preconditioned methods (Muon, SOAP) and a momentum‑scheduled method (ADANA) compared to AdamW. Across models ranging from 51M to 253M parameters and overtraining factors up to 256×, the authors find that optimal learning‑rate schedules, weight‑decay coefficients, and memory settings shift with horizon, and that ADANA consistently outperforms AdamW, especially with log‑time weight decay and momentum cooldown. Muon and SOAP maintain roughly constant token‑efficiency advantages, with SOAP potentially improving at the highest overtraining levels.
arXiv Machine Learning
Jun 3
MuLoCo: Muon is a practical inner optimizer for DiLoCo
arXiv:2505. 23725v3 Announce Type: replace Abstract: DiLoCo is a powerful framework for training large language models (LLMs), enabling larger optimal batch sizes and increased accelerator utilization under networking constraints.
arXiv AI
Jun 12
LoRA-Muon: Spectral Steepest Descent on the Low-Rank Manifold
arXiv:2606. 12921v1 Announce Type: cross Abstract: Low-Rank Adaptation (LoRA) significantly reduces compute and memory costs for finetuning Deep Learning models but is often harder to tune than dense training: when using factor-wise optimizers such as AdamW, it is sensitive to initialization choices, its optimal learning rates transfer poorly across ranks, and it often fails to beat dense baselines.