arXiv Computation and Language November 4, 2026 By Piyush Dhoka, Vedant Pandhare, Varun Inamdar, Aadarsh Pathre, Pavitha Nooji

What Breaks and What Comes Back: Capability Damage and Recovery in Depth-Pruned Small Language Models

Read the original on arXiv Computation and Language →

Depth pruning removes entire Transformer blocks to shrink and speed up language models, but the resulting damage is uneven across capabilities. In a study of four 1.2–1.7 B instruction‑tuned models, function calling and mathematical reasoning collapsed in every case, while other skills varied in order of degradation. Fine‑tuning a short ‘healing’ run can restore some abilities, with the choice of which blocks to retrain and the allocation of a limited token budget influencing recovery outcomes.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computation and Language.

arXiv Computation and Language

Sep 28

RAZOR: Pruning Replaceable Experts in LLMs

arXiv:2609.30465v1 Announce Type: cross Abstract: Mixture-of-experts (MoE) models activate few experts per token but store the full expert pool. Expert pruning reduces this storage burden; at a fixed...

By Mingyang Song, Mao Zheng

llms efficiency benchmarks

arXiv AI

Jun 30

Data and Evaluation Closed-Loop for Model Capability Enhancement

arXiv:2606. 28471v1 Announce Type: new Abstract: Model capability is the central variable in LLM pre-training, yet is never observed directly: data shapes it prospectively, while evaluation reveals it only retrospectively, compressing samples, prompts, decoding, and scoring rules into one noisy score.

By Zhixuan Li, Jiangan Yuan, Han Xu

llms benchmarks

arXiv Machine Learning

Jul 21

Half the Experts, All the Code: One-Shot Domain Pruning of Mixture-of-Experts LLMs for Coding

arXiv:2607. 16721v1 Announce Type: new Abstract: The strongest open-weight coding models are mixture-of-experts (MoE) networks: most of their size comes from large pools of "expert" subnetworks, of which only a few act on any token.

By Anik Jha

llms agents fine-tuning efficiency benchmarks

arXiv Machine Learning

Jul 31

The Kinetics of Training: A Driven-Nucleation Rate Law for Emergence, Plasticity Loss, and Circuit Control in Language Models

arXiv:2607. 27281v1 Announce Type: new Abstract: A capability appears in a language model when the last parts of its circuit align in one stochastic attempt, and getting all but one right is worth nothing.

By Lei Dong

llms safety

arXiv AI

Sep 10

Train Overcomplete, Deploy Compact: Scaling Recovery Capacity for Structured LLM Pruning

arXiv:2609.06974v1 Announce Type: cross Abstract: Large language models achieve strong performance across diverse tasks, but deployment remains costly because of memory, latency, and energy demands....

By Seungmin Oh, Donggeon Lee, Jongbin Ryu

llms efficiency

arXiv AI

Jul 16

ShortOPD: Recovering Pruned LLMs with Short-to-Long On-Policy Distillation

arXiv:2607. 13124v1 Announce Type: cross Abstract: Structured pruning is a hardware-friendly way to compress LLMs, but it is mostly validated on multiple-choice recognition tasks, while the same compressed checkpoints can collapse on the free-form generation that deployment actually requires.

By Qingyu Zhang, Qianhao Yuan, Hongyu Lin, Yaojie Lu, Xianpei Han, Le Sun, Xiang Li, Ming Xu, Jiarui Li, Xiuyin Zhao

llms fine-tuning efficiency benchmarks

One email a morning, machine-written

Related stories

RAZOR: Pruning Replaceable Experts in LLMs

Data and Evaluation Closed-Loop for Model Capability Enhancement

Half the Experts, All the Code: One-Shot Domain Pruning of Mixture-of-Experts LLMs for Coding

The Kinetics of Training: A Driven-Nucleation Rate Law for Emergence, Plasticity Loss, and Circuit Control in Language Models

Train Overcomplete, Deploy Compact: Scaling Recovery Capacity for Structured LLM Pruning

ShortOPD: Recovering Pruned LLMs with Short-to-Long On-Policy Distillation