← Back to all news
arXiv Machine Learning By Thomas Vaitses Fontanari, Maximo Eduardo Rulli, Federico Alvetreti, Donatella Genovese, Simone Scardapane

LRCC: Generalizing Low-Rank Compression with Conditional Computation

Read the original on arXiv Machine Learning →

The paper introduces Low‑Rank Conditional Computation (LRCC), a method that adds token‑dependent computation to pretrained language models by training lightweight routers to select among nested low‑rank paths while keeping low‑rank factors frozen. LRCC is evaluated on Llama and Qwen models, showing improved predictive performance over static low‑rank compression, including a 7.6‑percentage‑point gain in downstream accuracy on Llama‑2‑7B. The study also demonstrates that LRCC enhances perplexity and downstream accuracy on Llama‑3.2‑1B at matched decoding latency and analyzes the routers’ path choices to assess token‑wise path usefulness.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

Related stories

arXiv AI
Jul 21

Sparsity-Aware Low-Rank Representation for Efficient Fine-Tuning of Large Language Models

arXiv:2601. 16991v3 Announce Type: replace-cross Abstract: Adapting large pre-trained language models to downstream tasks often entails fine-tuning millions of parameters or deploying costly dense weight updates, which hinders their use in resource-constrained environments.

By Longteng Zhang, Sen Wu, Shuai Hou, Zhengyu Qing, Zhuo Zheng, Danning Ke, Qihong Lin, Qiang Wang, Shaohuai Shi, Xiaowen Chu
llmsfine-tuningefficiencybenchmarks
More like this →
arXiv AI
Aug 26

Compression Trinity: Exploring Sparsity, Quantization, and Low-Rank Approximations for LLM Compression

The paper introduces the "Compression Trinity," a unified framework that jointly applies sparsity, quantization, and low‑rank approximations to compress large language models. It presents several methods—MKOR, SLoPe, OPTIMA, PATCH, and SLiM—that leverage these three pillars to accelerate training, reduce memory bandwidth, and recover accuracy, achieving significant speedups and accuracy gains over existing techniques. The results demonstrate that combining all three compression strategies is essential for efficient, scalable, high‑performance LLM deployment.

By Mohammad Mozaffari
llmsfine-tuningefficiencybenchmarks
More like this →
arXiv Machine Learning
Sep 15

MoARa: Module-Aware Rank Allocation and Structure-Preserving Decomposition for Low-Rank LLM Pre-training

MoARa introduces a module-aware rank allocation strategy and a block-wise magnitude-direction decomposition to improve low-rank gradient projection for large language model pre‑training. By profiling Transformer modules and tailoring projection ranks, it reduces the number of steps and wall‑clock time needed to reach target perplexity. Experiments on Llama, Qwen, and DeepSeek models show up to 41.7% fewer steps and 37.1% less training time with minimal memory overhead.

By Keunyoung Kim, Nojun Kwak
llms
More like this →
arXiv Machine Learning
Aug 28

LowRankArena: A Standardized Evaluation Platform for SVD-Based LLM Compression

LowRankArena is a standardized evaluation platform for SVD‑based low‑rank compression of large language models, unifying task versions, compression budgets, comparison regimes, and inference measurements. It provides a reproducible pipeline with over 3 TiB of released compressed checkpoints, enabling consistent comparisons across methods. An audit of five representative SVD techniques using LowRankArena shows that prior reported gains are highly conditional, with performance leaders and tiers shifting across backbones and keep ratios, and that nominal low‑rank savings often yield limited end‑to‑end speedups.

By Zishan Shao, Lixun Zhang, Kangning Cui, Wenhao Wu, Jinhee Kim, Yixiao Wang, Ting Jiang, Hancheng Ye, Qinsi Wang, Fan Yang, Danyang Zhuo, Yiran Chen, Hai Li
llmsbenchmarks
More like this →