Z-Loss Backward Geometry in Dense Output Heads and Sparse Routers
Read the original on arXiv Machine Learning →The Flow has not summarised this story yet — read it at arXiv Machine Learning.
The Flow has not summarised this story yet — read it at arXiv Machine Learning.
arXiv:2609.37717v1 Announce Type: new Abstract: Decoder-only transformers are trained only through a terminal next-token prediction loss, yet this loss constrains every intermediate hidden state thro...
arXiv:2606. 05516v1 Announce Type: new Abstract: Zeroth-order (ZO) optimization enables memory-efficient fine-tuning of large language models (LLMs) using only forward passes, but it remains unclear how useful adaptation is distributed across layers.
arXiv:2606. 16112v1 Announce Type: cross Abstract: Residual architectures are ubiquitous in deep learning, but they suffer from a subtle structural limitation: the norm of the residual stream can grow rapidly with depth.
The paper investigates training‑free pruning of Mixture‑of‑Experts (MoE) models, focusing on how to handle the computation when a router selects a pruned expert. It compares replacement operators from three families on several models (OLMoE, DeepSeek‑V2‑Lite, Moonlight, Granite) while keeping the pruning mask fixed, and introduces a new operator called the tilted novice. The study finds that no single operator consistently dominates, with performance varying by model, pruning level, and evaluation metric, and it proposes a five‑item checklist for reporting results.
arXiv:2601. 04710v2 Announce Type: replace-cross Abstract: Fine-tuning large language models (LLMs) achieves strong performance but is often limited by the memory overhead of backpropagation.
arXiv:2607. 24665v1 Announce Type: cross Abstract: Modern large language models scale successfully by pairing capacity growth with efficiency, keeping per-token and deployment costs under control as capacity grows.