← Back to all news
arXiv Machine Learning By Estelle Zheng, S\'ebastien Warichet, Emmanuel Helbert, Christophe Cerisara

Learning New Facts with QLoRA: An Acquisition-Retention Frontier

Read the original on arXiv Machine Learning →

The paper investigates how parameter‑efficient fine‑tuning affects a language model’s ability to learn new facts while retaining existing knowledge. Using a controlled OpenStreetMap benchmark, the authors compare full fine‑tuning with quantized low‑rank adaptation (QLoRA) at various ranks, finding a clear trade‑off: lower ranks preserve out‑of‑domain performance but acquire fewer facts, whereas higher ranks improve factual generalization at the cost of unrelated benchmark performance. Diagnostics in distribution, weight space, and spectral analyses confirm this acquisition‑retention frontier, and a separate math adaptation experiment shows a weaker effect, indicating the phenomenon is strongest when installing new factual associations.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

Related stories

arXiv Computation and Language
Sep 1

When to Adapt: Conditional Memory Adapters for Retention-Preserving Domain Specialization

The paper introduces Engram Adapter, a conditional memory adapter that selectively activates on in-domain inputs while suppressing out-of-domain (OOD) retrievals. It leverages multi‑channel matching over local n‑gram patterns with occupancy tracking and a learned scalar gate to inject residuals only when appropriate. Experiments on Qwen3-4B/8B with AG‑News and MedMCQA show improved in‑domain accuracy while preserving nearly all OOD performance, outperforming always‑on baselines on LegalBench.

By Jiayu Hou, Lei Wang
llmsfine-tuningbenchmarks
More like this →
Hugging Face Trending Papers
Sep 28

Learn Here, Move Less Elsewhere: Input-Conditioned Plasticity from Retained-Domain Activation Atlases

The paper introduces ATLAS, a method that uses retained-domain activation atlases to guide task-specific fine‑tuning of language models. By providing local reference centers and directional filters, ATLAS learns a low‑rank residual that adapts to new tasks while preserving existing behavior. Experiments on Qwen3‑8B and other backbones show lower retained‑output KL divergence, fewer rewritten answers, and more stable responses compared to seven baselines.

llmsfine-tuning
More like this →
arXiv AI
5d ago

Where to Adapt Matters: Layer-Selective Fine-Tuning for Capability Retention

The paper investigates how the choice of Transformer layers for fine‑tuning affects both task performance and the preservation of pre‑trained capabilities. By measuring layer sensitivity with a forward‑only input–output cosine similarity metric, the authors identify layers that are more suitable for adaptation. They introduce Layer‑Selective LoRA (LS‑LoRA), which trains adapters only in low‑similarity layers, and demonstrate that this approach improves target‑task results while better retaining commonsense reasoning compared to standard all‑layer LoRA.

By Zhiqiang Pang, Zihong Sun, Qi Xie, Jun Shu, Deyu Meng, Zongben Xu
llmsfine-tuning
More like this →