arXiv:2609.37717v1 Announce Type: new
Abstract: Decoder-only transformers are trained only through a terminal next-token prediction loss, yet this loss constrains every intermediate hidden state thro...
By Timur Mudarisov, Mikhail Burtsev, Tatiana Petrova, Radu State
arXiv:2606. 05516v1 Announce Type: new Abstract: Zeroth-order (ZO) optimization enables memory-efficient fine-tuning of large language models (LLMs) using only forward passes, but it remains unclear how useful adaptation is distributed across layers.
By Wanhao Yu, Ziyan Wang, Zheng Wang, Abeer Matar Almalky, Yihang Zuo, Shuteng Niu, Sen Lin, Adnan Siraj Rakin, Deliang Fan, Li Yang
arXiv:2606. 16112v1 Announce Type: cross Abstract: Residual architectures are ubiquitous in deep learning, but they suffer from a subtle structural limitation: the norm of the residual stream can grow rapidly with depth.
By Tom\'as Figliolia, Beren Millidge
The paper investigates training‑free pruning of Mixture‑of‑Experts (MoE) models, focusing on how to handle the computation when a router selects a pruned expert. It compares replacement operators from three families on several models (OLMoE, DeepSeek‑V2‑Lite, Moonlight, Granite) while keeping the pruning mask fixed, and introduces a new operator called the tilted novice. The study finds that no single operator consistently dominates, with performance varying by model, pruning level, and evaluation metric, and it proposes a five‑item checklist for reporting results.
By Ezza Ali, Sehrish Bibi, Momina Moetesum, Faisal Shafait
arXiv:2601. 04710v2 Announce Type: replace-cross Abstract: Fine-tuning large language models (LLMs) achieves strong performance but is often limited by the memory overhead of backpropagation.
By Feihu Jin, Shipeng Cen, Ying Tan
arXiv:2607. 24665v1 Announce Type: cross Abstract: Modern large language models scale successfully by pairing capacity growth with efficiency, keeping per-token and deployment costs under control as capacity grows.
By Yanhao Jia, Jiepeng Wang, Haibin Huang, Chi Zhang, Erik Cambria, Xuelong Li
arXiv:2607. 13380v1 Announce Type: new Abstract: Predictive Coding (PC) offers a biologically motivated alternative to backpropagation via local weight updates, yet routing error between layers still relies on an autograd Jacobian-transpose ($J^\top$) product - the last non-local operation in PC.
By Junlong Shen, Xingyu Li
arXiv:2609.37899v1 Announce Type: new
Abstract: Zero-order optimization (ZO) trains without backpropagation, making it relevant to forward-only hardware and non-differentiable loss, but its gradient...
By Francois Chaubard, Mykel J. Kochenderfer, Chris R\'e
Softmax routing in mixture-of-experts models can concentrate probability on a subset of experts, suppressing access to the others. We study how this concentration forms, why it persists, and how suppr...
SHIFT-LLM is a training‑free post‑pruning correction framework that inserts a Linear Residual Adapter (LRA) at each depth‑pruned site in large language models. Each LRA preserves the original residual identity while adding a lightweight affine correction calibrated via closed‑form least‑squares regression on a small held‑out set, thereby approximating the hidden state that would have been produced by the removed block. Experiments across multiple model families and benchmarks show that SHIFT‑LLM consistently recovers accuracy lost to depth pruning, achieving gains up to +15.7 points on Llama‑3.1‑8B‑Instruct with only a few hundred calibration samples and no gradient computation.
By Ali Bahri, Hang Li, Hongliang Li, Zhitang Chen
The paper addresses training instabilities in large language model pretraining, specifically output logit divergence that occurs near the end of training. By analyzing the geometry of output embeddings, the authors identify anisotropic embeddings as the root cause and propose Output Embedding Centering (OEC) as a mitigation strategy. OEC can be applied deterministically as μ‑centering or as a regularization loss μ‑loss, and experiments show both variants outperform the existing z‑loss method while matching logit soft‑capping in stability, even without weight tying. Additionally, μ‑loss is less sensitive to hyperparameter tuning than z‑loss.
By Felix Stollenwerk, Anna Lokrantz, Niclas Hertzberg
The paper examines whether expert‑routing statistics from a Mixture‑of‑Experts language model can inform layer‑wise mixed‑precision quantization of routed‑expert feed‑forward networks. By building a routing‑capture pipeline for Qwen‑3.6‑35B‑A3B, the authors derive a composite sensitivity score and use coordinate descent to allocate bits per layer under a fixed budget. Experiments across flat, steep, and random allocation regimes show that routing‑guided reallocation does not outperform a uniform baseline, suggesting that routing behavior is an unreliable proxy for true quantization cost.
By Yifan Liu