arXiv AI
Depth and Scale in the Sub-150M Regime: JugnuLM-53M vs JugnuLM-110M
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv Machine Learning
Sep 3
Train What You Deploy: Closing the MLP Reachability Gap in Low-Rank Clone Distillation
The paper introduces a new approach to low‑rank clone distillation that ensures the student model’s training targets exactly the weights it will use at inference. By redefining the training objective to cover the full deployed matrix—without changing the model’s shape, parameter count, or FLOPs—the authors recover previously unreachable linear degrees of freedom. This results in significant performance gains across multiple teacher models, achieving comparable or superior accuracy with fewer tokens and parameters.
arXiv Computation and Language
3d ago
Tune the Baseline First: Most of Muon's Scaled Weight Decay Gain Is a Tuning Gap, but Its Small Real Advantage Grows With Model Size
The study re‑examines scaled weight decay (Muon‑SW) versus constant decoupled weight decay for language models, tuning learning‑rate and weight‑decay jointly across a range of model sizes. It finds that most of the reported performance gain is due to a tuning gap, with the tuned advantage shrinking to only 0.0026 BPB. However, the remaining advantage grows with model size, rising from 0.0036 to 0.0051 BPB over an eight‑fold increase in active parameters, and the optimal LR‑WD product shifts faster for Muon‑SW.
arXiv Machine Learning
Aug 5
Wiring Beats Blending: What Transfers Between Transformer Sizes -- and What Doesn't
arXiv:2608. 02829v1 Announce Type: new Abstract: Model families train every size from scratch.
arXiv AI
Sep 25
Wiring Beats Blending: Structure-Aware Compensation for Transformer Downscaling
The paper investigates converting a large pretrained transformer (1.4 B parameters) into a smaller sibling (410 M) by studying representation alignment and parameter projection. It finds that dense weight projection destroys structure, and that a low‑budget, structure‑aware compensation—separating least‑squares function alignment from variance‑preserving rescaling—yields significant gains on token‑efficient training, outperforming subcloning and standard distillation pipelines at matched budgets.
arXiv Machine Learning
1d ago
FastAFD: When Does Attention-FFN Disaggregation Pay on Rack-Scale NVLink?
FastAFD is an attention‑FFN disaggregation engine designed for rack‑scale NVLink systems such as GB200 NVL72. It overlaps the mixture‑of‑experts computation of one microbatch with the attention of another, achieving up to 1.17× higher throughput than the best wide‑expert‑parallelism vLLM configuration on Qwen3 and Qwen3.5, and matching it on MiniMax‑M2.5. An analytical model predicts AFD performance within 3% using only colocated vLLM profiles, revealing that speedup comes from batch and overlap gains that outweigh disaggregation costs, and that speculative decoding further boosts performance for light‑expert models.
arXiv Computation and Language
Sep 25
Baseline Shape Decides the Verdict: A Controlled Re-Examination of Ternary Language Models at 60K Parameters
The study re‑examines a reported advantage of a routed ternary (1.58‑bit) language model over a full‑precision transformer at 60K parameters. By running controlled experiments with multiple seeds and a fixed training recipe, the authors find that the apparent benefit largely stems from the choice of baseline model shape rather than the ternary architecture itself. While the routed model does outperform other shapes at a larger 130M‑byte budget, its advantage diminishes when a plain gated diagonal‑SSM block is used, and the ternary penalty varies with architecture and quantization details.