The paper introduces DKL, a method for adding new knowledge to instruction‑tuned language models without compromising their instruction‑following abilities. DKL performs extended pre‑training on a base LLM to embed knowledge, then merges these weights into the instruction‑tuned model, avoiding costly instruction fine‑tuning. Experiments show DKL raises RAG accuracy from 54.17% to 79.26% on retrieval failure cases while using far less training data than previous approaches.
By Kushagra Bhushan, Meghanadh Pulivarthi, Sai Krishna Reddy Sathi, Gaurav Pandey, Sonam Gupta, Vineet Kumar, Jaydeep Sen, Yatin Nandwani, Sachindra Joshi, Dinesh Raghu
arXiv:2609.37891v1 Announce Type: cross
Abstract: Current pre-training datasets are derived from web crawls, with all their issues, and were not designed to support mid- and post-training pipelines--...
By Pierre-Carl Langlais, Pieter Delobelle, Yannick Detrois, Pavel Chizhov, Carlos Rosas-Hinostroza, Neil Si Smail, Benjamin Burtin, Hanna Shcharbakova, Ivan Yamshchikov, Anastasia Stasenko
The paper introduces DKL, a method that decouples knowledge learning from instruction tuning in language models. Instead of fine‑tuning the instruction‑tuned model directly, DKL first extends pre‑training on a base model to embed new knowledge, then merges these weights into the instruction‑tuned model, preserving its instruction‑following abilities. Experiments show DKL raises RAG accuracy from 54.17 % to 79.26 % on retrieval failures, outperforming prior methods while using far less training data.
arXiv:2601. 22146v2 Announce Type: replace-cross Abstract: Due to limited supervised training data, large language models (LLMs) are typically pre-trained via a self-supervised "predict the next word" objective on a vast amount of unstructured text data.
By Ajay Patel, Colin Raffel, Chris Callison-Burch
arXiv:2608.30627v1 Announce Type: new
Abstract: As language-model compute continues to scale, high-quality training data is becoming an increasingly important bottleneck. Conventional next-token pred...
By Haoran Que, Jiajun Shi, Ting Huang, Renming Pang, Jiaheng Liu, Ge Zhang, Wenhao Huang, Shen Yan, Wei Ye, Shikun Zhang
arXiv:2608. 20281v1 Announce Type: cross Abstract: Large language models often fail to answer questions about a bounded document collection when the source documents are not retrieved at inference time.
By Qian Kou, Xiaofeng Shi, Xiaosong Qiu, Hua Zhou
The paper introduces a data‑centric pipeline for post‑training language models on financial reasoning tasks. It mines open‑source reasoning traces, distills financial instruction data, and generates knowledge‑graph‑guided question‑answer pairs, then filters examples with lightweight classifiers and applies reinforcement learning with rule‑based verifiers. Experiments on FINESSE‑Bench show that retention‑aware adaptation—self‑distilled fine‑tuning and model merging—outperforms ordinary supervised fine‑tuning, improving accuracy by up to 3.0 points and avoiding regressions.
By Zhirayr Hayrapetyan, Andrei Kalmykov, Denis Kokosinskii, Dmitry Stanishevskii, Dmitry Zmitrovich
arXiv:2510.13935v3 Announce Type: replace-cross
Abstract: The facts a language model stores are tied to its parameter count, so small models that fit on edge devices fail on expert problems, which ne...
By Kenan Alkiek, David Jurgens, Vinod Vydiswaran
arXiv:2609.35868v1 Announce Type: new
Abstract: Is human readability necessary for effective fine-tuning of large language models? We investigate whether model-conditioned training representations ca...
By Jinhao Zhang, Zeyu Liu, Zicheng Yan, Yunquan Zhang, Daning Cheng, Song Tang
The paper introduces GONE, a benchmark for evaluating knowledge unlearning in large language models using structured knowledge graphs, and presents Neighborhood-Expanded Distribution Shaping (NEDS), a framework that leverages graph connectivity to separate forgotten facts from their semantic neighborhood. GONE disentangles direct fact removal, reasoning-based leakage, and catastrophic forgetting, while NEDS achieves high unlearning efficacy and locality on LLaMA-3-8B and Mistral-7B. The dataset is publicly available on Hugging Face.
By Chahana Dahal, Ashutosh Balasubramaniam, Zuobin Xiong
arXiv:2606. 26807v1 Announce Type: new Abstract: We propose a new method that allows an LLM to automatically pull in factual knowledge from a knowledge base during token generation.
By Francois Crespin (IP Paris, LTCI), Fabian M. Suchanek (IP Paris, LTCI), Nils Holzenberger
arXiv:2609.17019v1 Announce Type: new
Abstract: While Chain-of-Thought (CoT) reasoning has been proven to be effective, it often leads to overthinking, resulting in computational overhead, inference...
By Qinhong Lin, Yuhao Zhang, Yinglun Feng, Zhongliang Yang, Linna Zhou