Low-rank adaptation fixes the rank of the update, but it does not identify which parts of a trained
write actually carry behavior. We study that question directly and show that behaviorally effectiv...
The paper introduces READ, a method for composing low‑rank adapters (LoRA) in large language models. By rewriting each adapter into a balanced canonical form and enforcing a one‑directional coupling, READ allows new skills to read but never write into the output subspaces of existing skills, eliminating interference. Experiments on four benchmark suites and two model families show that READ consistently outperforms existing baselines, improving SuperGLUE scores by over twenty points and domain suite scores by more than seven points.
By Zeyan Li, Panqi Yang, Qirong Guo, Shengda Zhuo, SIyuan Qiu, Hu Xu, Chun Li, Jianfeng Xu
Federated fine-tuning of large pre-trained models increasingly relies on Low-Rank Adaptation (LoRA) to reduce communication and computation, but heterogeneous clients can make adapter aggregation unstable. We identify the data-parameter interference as a geometric source of this instability.
arXiv:2607. 14367v1 Announce Type: new Abstract: Federated fine-tuning of large pre-trained models increasingly relies on Low-Rank Adaptation (LoRA) to reduce communication and computation, but heterogeneous clients can make adapter aggregation unstable.
By Haobo Zhang, Jiankun Wang, Suraj Rajendran, Weishen Pan, Lam Tsoi, Yong Chen, Fei Wang, Jiayu Zhou
Delta N-gram introduces a forward‑only write path into a language model’s native hashed N‑gram table, preserving the pretrained table while adding evidence through a sparse patch Δ. The method uses a 4.1 M‑parameter writer that emits gated targets per event and applies a block‑averaged delta rule to keep each row a convex combination of its old value and the new targets. Experiments on a 0.6 B prototype and Qwen3.8‑Flash‑Next show that the delta writer maintains more document‑specific memory and reduces drift compared to online gradient writes, achieving stable performance over thousands of writes.
whyItMatters":"The approach allows continual updates to a language model’s memory without altering its pretrained parameters, enabling efficient, stable, and document‑specific knowledge retention during inference."
By Hao Wu
arXiv:2605.06183v2 Announce Type: replace
Abstract: Low-rank adaptation (LoRA) is a widely used parameter-efficient fine-tuning method that places trainable low-rank adapters into frozen pre-trained...
By Suoxin Zhang, Run He, Di Fang, Xiang Tan, Kaixuan Chen, Huiping Zhuang