The paper introduces DKL, a method that decouples knowledge learning from instruction tuning in language models. Instead of fine‑tuning the instruction‑tuned model directly, DKL first extends pre‑training on a base model to embed new knowledge, then merges these weights into the instruction‑tuned model, preserving its instruction‑following abilities. Experiments show DKL raises RAG accuracy from 54.17 % to 79.26 % on retrieval failures, outperforming prior methods while using far less training data.
arXiv:2601. 22146v2 Announce Type: replace-cross Abstract: Due to limited supervised training data, large language models (LLMs) are typically pre-trained via a self-supervised "predict the next word" objective on a vast amount of unstructured text data.
By Ajay Patel, Colin Raffel, Chris Callison-Burch
The paper introduces Universal Textual Teaching (UTT), a parameter‑update‑free framework that distills knowledge gaps between a Teacher and Student large language model into a reusable natural‑language artifact called a Primer. UTT identifies representative gap cases, then iteratively refines the Primer through a cycle of student attempts, prompter‑generated teaching instructions, teacher demonstrations, and synthesizer consolidation. Experiments on math (Omni‑MATH‑2) and code generation (KernelBench) show significant accuracy gains, outperforming prompt engineering and parameter‑based knowledge distillation, and the Primer generalizes across different Teacher‑Student pairs.
By Zhanyi Lu, Huan Wang
arXiv:2510.13935v3 Announce Type: replace-cross
Abstract: The facts a language model stores are tied to its parameter count, so small models that fit on edge devices fail on expert problems, which ne...
By Kenan Alkiek, David Jurgens, Vinod Vydiswaran
The paper introduces Style‑Debiased DPO (SD‑DPO), a method that refines large language models’ ability to retrieve stored knowledge by using preference optimization that corrects for style differences while preserving factual accuracy. SD‑DPO evaluates on the EntiGraph storing‑side framework and outperforms baseline CPT on the QuALITY reading‑comprehension benchmark, achieving higher accuracy with far fewer training tokens. In a knowledge‑editing setting (AToKE), SD‑DPO attains an overall accuracy of 0.982, correctly answering queries with either new or old facts based on the requested time period.
By Takayuki Yamamoto, Daisuke Kawahara
arXiv:2608. 20281v1 Announce Type: cross Abstract: Large language models often fail to answer questions about a bounded document collection when the source documents are not retrieved at inference time.
By Qian Kou, Xiaofeng Shi, Xiaosong Qiu, Hua Zhou