arXiv Computation and Language September 2, 2026 By Haruto Sato, Yuki Tanaka, Ren Nakamura, Aoi Kobayashi, Mei Ito

Behaviorally Effective LoRA Writes Are Sparse and Structured

Read the original on arXiv Computation and Language →

The paper investigates which parts of a low‑rank adaptation (LoRA) update actually drive behavior in language models. It shows that behaviorally effective LoRA writes are sparse, structured, and highly concentrated, with only a few module‑wise components carrying most of the signal. Using a learned‑basis continuation recipe, the authors demonstrate that these sparse, structured writes can be recovered with minimal error and that they remain robust across different training continuations and evaluation tasks.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computation and Language.

fine-tuning

Hugging Face Trending Papers

Sep 1

Behaviorally Effective LoRA Writes Are Sparse and Structured

Low-rank adaptation fixes the rank of the update, but it does not identify which parts of a trained write actually carry behavior. We study that question directly and show that behaviorally effectiv...

fine-tuning

Hugging Face Trending Papers

Jul 15

Dysco: Dynamic Subspace Boosting to Mitigate LoRA Interference in Federated Learning

Federated fine-tuning of large pre-trained models increasingly relies on Low-Rank Adaptation (LoRA) to reduce communication and computation, but heterogeneous clients can make adapter aggregation unstable. We identify the data-parameter interference as a geometric source of this instability.

llms fine-tuning safety

arXiv Machine Learning

Jul 17

Dysco: Dynamic Subspace Boosting to Mitigate LoRA Interference in Federated Learning

arXiv:2607. 14367v1 Announce Type: new Abstract: Federated fine-tuning of large pre-trained models increasingly relies on Low-Rank Adaptation (LoRA) to reduce communication and computation, but heterogeneous clients can make adapter aggregation unstable.

By Haobo Zhang, Jiankun Wang, Suraj Rajendran, Weishen Pan, Lam Tsoi, Yong Chen, Fei Wang, Jiayu Zhou

llms fine-tuning safety

arXiv AI

Jul 20

PASs-MoE: Mitigating Misaligned Co-drift among Router and Experts via Pathway Activation Subspaces for Continual Learning

arXiv:2601. 13020v2 Announce Type: replace-cross Abstract: Continual instruction tuning (CIT) requires multimodal large language models (MLLMs) to adapt to a stream of tasks without forgetting prior capabilities.

By Zhiyan Hou, Haiyun Guo, Haokai Ma, Yandu Sun, Yonghui Yang, Jinqiao Wang

llms fine-tuning multimodal benchmarks

arXiv AI

Jun 15

Beyond LoRA: Is Sparsity-Induced Adaptation Better?

arXiv:2606. 13767v1 Announce Type: cross Abstract: Low-rank adaptation (LoRA) and its variants provide a memory- and compute-efficient alternative to full fine-tuning of pre-trained models.

By Elijah Cadenhead, Cristian McGee, Xin Li, El Houcine Bergou, Aritra Dutta

fine-tuning efficiency

arXiv Machine Learning

Aug 12

Same Targets, Different Computation: How Post-Training Divides Work Across Model Layers

arXiv:2605. 07284v2 Announce Type: replace Abstract: A late-layer change learned during post-training may work on the base model's earlier state, or it may depend on earlier computation learned with it.

By Yifan Zhou

llms fine-tuning

One email a morning, machine-written

Related stories

Behaviorally Effective LoRA Writes Are Sparse and Structured

Dysco: Dynamic Subspace Boosting to Mitigate LoRA Interference in Federated Learning

Dysco: Dynamic Subspace Boosting to Mitigate LoRA Interference in Federated Learning

PASs-MoE: Mitigating Misaligned Co-drift among Router and Experts via Pathway Activation Subspaces for Continual Learning

Beyond LoRA: Is Sparsity-Induced Adaptation Better?

Same Targets, Different Computation: How Post-Training Divides Work Across Model Layers