SkipCat introduces a rank‑maximized low‑rank compression framework for large language models, combining intra‑layer shared low‑rank projections and block‑skipping techniques to reduce redundancy and computational overhead. By allowing higher retained ranks while maintaining the same compression rate, SkipCat achieves a 7% accuracy improvement on zero‑shot tasks compared to prior low‑rank methods, all without additional fine‑tuning.
By Yu-Chen Lu, Sheng-Feng Yu, Hui-Hsien Weng, Pei-Shuo Wang, Yu-Fang Hu, Liang Hung-Chun, Hung-Yueh Chiang, Kai-Chiang Wu
arXiv:2601. 16991v3 Announce Type: replace-cross Abstract: Adapting large pre-trained language models to downstream tasks often entails fine-tuning millions of parameters or deploying costly dense weight updates, which hinders their use in resource-constrained environments.
By Longteng Zhang, Sen Wu, Shuai Hou, Zhengyu Qing, Zhuo Zheng, Danning Ke, Qihong Lin, Qiang Wang, Shaohuai Shi, Xiaowen Chu
The paper introduces Low‑Rank Conditional Computation (LRCC), a method that adds token‑dependent computation to pretrained language models by training lightweight routers to select among nested low‑rank paths while keeping low‑rank factors frozen. LRCC is evaluated on Llama and Qwen models, showing improved predictive performance over static low‑rank compression, including a 7.6‑percentage‑point gain in downstream accuracy on Llama‑2‑7B. The study also demonstrates that LRCC enhances perplexity and downstream accuracy on Llama‑3.2‑1B at matched decoding latency and analyzes the routers’ path choices to assess token‑wise path usefulness.
By Thomas Vaitses Fontanari, Maximo Eduardo Rulli, Federico Alvetreti, Donatella Genovese, Simone Scardapane
The paper introduces 3BASiL-TM, a one‑shot post‑training method for compressing large language models using a sparse plus low‑rank decomposition. It employs a novel 3‑block ADMM algorithm to minimize layer‑wise reconstruction error and a transformer‑matching refinement that jointly optimizes sparse and low‑rank components across layers. Experiments show a >30% reduction in WikiText2 perplexity gap for LLaMA‑8B under a (2:4 sparse + 64 LR) setting and a 2.5× faster compression runtime on an A100 GPU compared to state‑of‑the‑art methods.
By Mehdi Makni, Xiang Meng, Rahul Mazumder
SHIFT-LLM is a training‑free post‑pruning correction framework that inserts a Linear Residual Adapter (LRA) at each depth‑pruned site in large language models. Each LRA preserves the original residual identity while adding a lightweight affine correction calibrated via closed‑form least‑squares regression on a small held‑out set, thereby approximating the hidden state that would have been produced by the removed block. Experiments across multiple model families and benchmarks show that SHIFT‑LLM consistently recovers accuracy lost to depth pruning, achieving gains up to +15.7 points on Llama‑3.1‑8B‑Instruct with only a few hundred calibration samples and no gradient computation.
By Ali Bahri, Hang Li, Hongliang Li, Zhitang Chen
The paper introduces REP‑LIE, a resource‑efficient pruning method for Transformer models that estimates weight importance using gradients from LoRA low‑rank matrices, avoiding full gradient computation. It incorporates a stability score to iteratively prune unimportant parameters and then fine‑tunes the pruned model with lightweight updates, eliminating the need for full‑parameter optimization. Experiments on medium‑scale encoders and large‑scale generative models such as LLaMA‑7B and Mistral‑7B show that REP‑LIE achieves competitive performance compared to existing pruning approaches.
By Peng Liu, Huibing Zeng, Yiqun Zhang, Yang Yi, Jigang Wu
With the rapid development of large-scale pre-trained language models based on Transformer architectures, their high computational and memory costs have become a major obstacle to deployment, especial...
Clean is a memory‑efficient optimizer that uses Nyström sketching to approximate SOAP’s curvature preconditioners, reducing memory complexity from quadratic to linear while preserving rich curvature information. It also reintegrates off‑subspace components and offers a low‑precision variant, Q‑Clean, that cuts optimizer memory by over 50% compared to Muon while maintaining strong performance. Clean outperforms AdamW in wall‑clock time and enables pre‑training a 13B‑parameter model on a single 80GB GPU.
By Beheshteh T. Rakhshan, Sahar Rajabi, Maziar Sargordi, Shikai Fang, Guillaume Rabusseau, Sirisha Rambhatla
arXiv:2607. 22251v1 Announce Type: new Abstract: Low-Rank Adaptation (LoRA) is a widely used parameter-efficient fine-tuning method for large language models, but its performance depends strongly on how a fixed rank budget is distributed across Transformer modules.
By Wei Zhang, Xinwu Liu, Yihang Cheng
Modern transformers pair impressive capabilities with substantial memory and compute demands. Low-rank weight factorization reduces both while keeping the matrices dense, and thus efficient on standar...
arXiv:2609.40127v1 Announce Type: cross
Abstract: Modern transformers pair impressive capabilities with substantial memory and compute demands. Low-rank weight factorization reduces both while keepin...
By Massimo Bini, Anders Christensen, Stephan Alaniz, Judah Goldfeder, Ole Winther, Yann LeCun, Ravid Shwartz-Ziv, Zeynep Akata
arXiv:2606. 27449v1 Announce Type: new Abstract: Multi-head attention conventionally partitions the hidden dimension equally across all heads at every layer, enforcing an identical representational subspace dimension (dh = dmodel/h) throughout the models depth.
By Shubham Aggarwal