← Back to all news
arXiv AI By Irene Tenison, Stella Ahn, Miriam Kim, Ebtisam Alshehri, Lalana Kagal

Activation-Efficient Fine-Tuning via Sequence-Compressed Low-Rank Subspaces

Read the original on arXiv AI →

The paper introduces LARS (Low-memory Activation-Rank Subspace), a parameter-efficient fine-tuning method that reduces the memory footprint of adapter activations by operating in a sequence-level low-rank subspace. By replacing token-level activations with pooled feature extraction, low-rank modulations, and residual projection, LARS cuts the stored activation intermediates from O(BSRL) to O(BRL). Experiments show that LARS achieves comparable performance to LoRA while lowering GPU memory usage by 23%, CPU memory usage by 47.8%, and reducing CPU power and energy consumption per optimization step by about 22%.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

Related stories

arXiv Machine Learning
Oct 15

The COUNT: Counting N-grams Unlocks Scaling on Host Memory and Training-Free Adaptation

The paper introduces COUNT, a method that builds n‑gram embeddings without adding learned parameters by storing precomputed n‑gram counts on host memory. By weighting the language model’s own unembeddings with corpus frequencies, COUNT can be added as an extra input embedding during pretraining, improving validation loss equivalent to a 57% increase in compute for a 1B‑parameter model. Additionally, COUNT enables efficient GPU memory usage and allows domain‑specific n‑gram tables to be swapped in at inference time for training‑free adaptation.

By Ian Magnusson, Noah A. Smith, Pang Wei Koh
llmsrag
More like this →
arXiv AI
Jul 21

Sparsity-Aware Low-Rank Representation for Efficient Fine-Tuning of Large Language Models

arXiv:2601. 16991v3 Announce Type: replace-cross Abstract: Adapting large pre-trained language models to downstream tasks often entails fine-tuning millions of parameters or deploying costly dense weight updates, which hinders their use in resource-constrained environments.

By Longteng Zhang, Sen Wu, Shuai Hou, Zhengyu Qing, Zhuo Zheng, Danning Ke, Qihong Lin, Qiang Wang, Shaohuai Shi, Xiaowen Chu
llmsfine-tuningefficiencybenchmarks
More like this →
arXiv Machine Learning
Aug 27

Ladder Up, Memory Down: Low-Cost Fine-Tuning With Side Nets

The paper introduces Ladder Side Tuning (LST), a parameter‑efficient fine‑tuning method that adds a lightweight side network to large language models. LST matches QLoRA’s compute scaling while halving peak memory usage, enabling 7B‑parameter models to be fine‑tuned on a single 12 GB GPU with 2k‑token contexts without gradient checkpointing. The authors also present xLadder, a depth‑extended variant that increases effective depth through cross‑connections, allowing deeper reasoning without extra memory overhead.

By Estelle Zheng, Nathan Cerisara, S\'ebastien Warichet, Emmanuel Helbert, Christophe Cerisara
llmsfine-tuningbenchmarks
More like this →