The paper investigates how parameter‑efficient fine‑tuning affects a language model’s ability to learn new facts while retaining existing knowledge. Using a controlled OpenStreetMap benchmark, the authors compare full fine‑tuning with quantized low‑rank adaptation (QLoRA) at various ranks, finding a clear trade‑off: lower ranks preserve out‑of‑domain performance but acquire fewer facts, whereas higher ranks improve factual generalization at the cost of unrelated benchmark performance. Diagnostics in distribution, weight space, and spectral analyses confirm this acquisition‑retention frontier, and a separate math adaptation experiment shows a weaker effect, indicating the phenomenon is strongest when installing new factual associations.
arXiv:2605.07111v3 Announce Type: replace-cross Abstract: Recent literature on fine-tuning Large Language Models highlights a fundamental debate. While Full Fine-Tuning (FFT) provides greater represe...
arXiv:2604. 00715v2 Announce Type: replace-cross Abstract: Retrieval-augmented generation (RAG) improves language model (LM) performance by providing relevant context at test time for knowledge-intensive situations.
The paper introduces Engram Adapter, a conditional memory adapter that selectively activates on in-domain inputs while suppressing out-of-domain (OOD) retrievals. It leverages multi‑channel matching over local n‑gram patterns with occupancy tracking and a learned scalar gate to inject residuals only when appropriate. Experiments on Qwen3-4B/8B with AG‑News and MedMCQA show improved in‑domain accuracy while preserving nearly all OOD performance, outperforming always‑on baselines on LegalBench.
The paper introduces ATLAS, a method that uses retained-domain activation atlases to guide task-specific fine‑tuning of language models. By providing local reference centers and directional filters, ATLAS learns a low‑rank residual that adapts to new tasks while preserving existing behavior. Experiments on Qwen3‑8B and other backbones show lower retained‑output KL divergence, fewer rewritten answers, and more stable responses compared to seven baselines.
The paper investigates how the choice of Transformer layers for fine‑tuning affects both task performance and the preservation of pre‑trained capabilities. By measuring layer sensitivity with a forward‑only input–output cosine similarity metric, the authors identify layers that are more suitable for adaptation. They introduce Layer‑Selective LoRA (LS‑LoRA), which trains adapters only in low‑similarity layers, and demonstrate that this approach improves target‑task results while better retaining commonsense reasoning compared to standard all‑layer LoRA.
arXiv:2608. 05326v1 Announce Type: new Abstract: Autoregressive large language model inference is increasingly constrained by the memory footprint of the Key-Value (KV) cache.
arXiv:2505. 18877v4 Announce Type: replace Abstract: Low-Rank Adaptation (LoRA) lowers the computational and memory overhead of fine-tuning large models by updating a low-dimensional subspace of the pre-trained weight matrix.
The paper introduces “PACE”, a training‑free framework that tackles bottlenecks in Retrieval‑Augmented Generation by frontloading evidence and adaptively budgeting reranking. It first reorders candidate documents based on marginal evidence coverage—prioritizing query‑relevant, complementary, and chain‑forming documents—providing a $(1-1/e)$ approximation guarantee. Then it dynamically adjusts the reranking budget according to the relative pressure of the reranker and the language model, improving evidence recall and reducing p95 latency in multi‑hop QA workloads.
arXiv:2609.05885v1 Announce Type: new Abstract: Low-rank adaptation (LoRA) has become the standard for parameter-efficient fine-tuning of large language models. Most LoRA variants follow a uniform-LR...
Low-rank adaptation (LoRA) has become a widely used parameter-efficient fine-tuning method for large language models. Since different modules and layers may contribute unequally to downstream adaptation, allocating rank resources under a fixed parameter budget is an important problem for balancing efficiency, expressiveness, and generalization.
arXiv:2607. 20205v1 Announce Type: cross Abstract: Low-rank adaptation (LoRA) has become a widely used parameter-efficient fine-tuning method for large language models.
arXiv:2608.28809v1 Announce Type: new Abstract: Extreme low-bit inference offers a route toward smaller models and constrained deployment. Ternary language models restrict weights to $\{-1,0,+1\}$, a...