arXiv Machine Learning September 30, 2026 By Tianhao Qian, Guilin Qi, Jiayu Chen

Relative Kinetic Utility: Calibrating Cross-Layer Credit for Global Structured LLM Pruning

Read the original on arXiv Machine Learning →

The paper introduces Global Relative Kinetic Utility (Global RKU), a label‑free method for calibrating cross‑layer credit in global structured pruning of large language models. Global RKU estimates channel importance via a final‑hidden‑state activation‑gradient signal and applies block‑relative normalization to remove block‑common scale while preserving within‑block ordering, enabling a single‑stage static pruning topology. Experiments on Qwen‑2.5‑7B show significant performance gains at various sparsity levels, and ablation studies confirm the effectiveness of the relative‑normalization step.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

arXiv Computer Vision

Sep 17

Unified Response Geometry for Structured Pruning

arXiv:2609.18239v1 Announce Type: new Abstract: Structured pruning is commonly formulated as ranking individual channels, although channel responses can be complementary or cancel through downstream...

By Kaixiang Shu

fine-tuning efficiency

arXiv Machine Learning

Sep 11

LILA: Calibration-Free Structured Pruning of Large Language Models via Latent Spectral Geometry

LILA (Latent-Informed Layer Analysis) introduces a calibration‑free method for structured pruning of large language models by scoring neuron importance using the Kolmogorov–Smirnov distance between singular value distributions of full and neuron‑ablated feed‑forward network weight matrices. The approach requires no training, calibration data, or auxiliary networks, and outperforms existing methods such as PruneNet and SliceGPT on LLaMA‑2‑7B and Phi‑2 at various sparsity levels. After a single epoch of LoRA fine‑tuning, LILA matches heavily calibrated baselines, and a Neural Tangent Kernel analysis provides theoretical support for its spectral importance criterion. Additionally, LILA can dynamically allocate sparsity budgets, achieving state‑of‑the‑art generative preservation and revealing architectural bottlenecks at higher compression.

By Sankar Behera, Dhruv Singh, Anshika Agnihotri, Raj Kumar Choudhary, Satyadev Ahlawat, Yamuna Prasad

llms reinforcement-learning fine-tuning efficiency benchmarks

arXiv AI

Jul 28

TriSP: Tri-Signal Structured Pruning for Large Language Models

arXiv:2607. 22587v1 Announce Type: new Abstract: Large language models (LLMs) achieve strong performance across diverse tasks but their deployment is constrained by the memory and compute cost of their parameters.

By Manel Kara laoua, Soumia Bouyahiaoui, Aicha Boutorh

llms fine-tuning efficiency

arXiv Computation and Language

5d ago

Beyond Mean Attention: Diversity-Aware, Layer-Wise Scoring for KV Cache Eviction

arXiv:2609.30738v1 Announce Type: new Abstract: KV cache eviction methods such as SnapKV and PyramidKV rank tokens solely by mean attention over a small observation window. We study a unified score,...

By Tianfang Xie, Wei Zhu

llms efficiency

arXiv Machine Learning

Jul 31

Averaged Evaluation Masks Capability Trade-Offs: Multi-Source Calibration for High-Sparsity LLM Pruning

arXiv:2606. 03328v3 Announce Type: replace Abstract: Calibration data are often treated as a minor implementation detail in post-training LLM pruning because averaged evaluations suggest only modest effects.

By Hu Xu, Zhaolong Xing, Congcong Liu, Jiaxing Wang, Zhida Jiang, Junshi Huang, Zhen Chen, Jianfeng Xu

llms efficiency

arXiv AI

Jul 7

HiFA4: Training-Free 4-bit FlashAttention on Ascend HIF4 NPUs for LLM Inference

arXiv:2607. 04302v1 Announce Type: cross Abstract: We present HiFA4, a post-training operator-level design that executes both QK^T and PV in FlashAttention as 4-bit HIF4 Cube GEMMs for LLM inference on Ascend NPUs, while maintaining the online softmax state in FP16.

By Hui Dong, Yanzhao Li, Jie Gao, Chunlu Li, Zhiyuan Zhang, Yupeng Sun, Zhenyuan Chen, Zhiqiang Zou

llms nlp efficiency benchmarks