The paper introduces DKL, a method for adding new knowledge to instruction‑tuned language models without compromising their instruction‑following abilities. DKL performs extended pre‑training on a base LLM to embed knowledge, then merges these weights into the instruction‑tuned model, avoiding costly instruction fine‑tuning. Experiments show DKL raises RAG accuracy from 54.17% to 79.26% on retrieval failure cases while using far less training data than previous approaches.
By Kushagra Bhushan, Meghanadh Pulivarthi, Sai Krishna Reddy Sathi, Gaurav Pandey, Sonam Gupta, Vineet Kumar, Jaydeep Sen, Yatin Nandwani, Sachindra Joshi, Dinesh Raghu
The paper introduces DKL, a method that decouples knowledge learning from instruction tuning in language models. Instead of fine‑tuning the instruction‑tuned model directly, DKL first extends pre‑training on a base model to embed new knowledge, then merges these weights into the instruction‑tuned model, preserving its instruction‑following abilities. Experiments show DKL raises RAG accuracy from 54.17 % to 79.26 % on retrieval failures, outperforming prior methods while using far less training data.
The paper investigates whether using a language model’s memory to personalize responses aligns with user preferences. By collecting about 1,000 pairwise judgments from 87 participants on ChatGPT queries that included personal memories, the authors find that memory‑conditioned responses are not consistently preferred and that users sometimes strongly favor non‑memory responses. An automatic reward model trained on these judgments achieves 72.8% preference prediction accuracy and outperforms existing baselines, highlighting the need for evaluation and optimization objectives that better reflect user preferences for memory use.
By Yijiang River Dong, Zexue He, Cheng Yu, Tiancheng Hu, Michael J Ryan, Jan Batzner, Zheng Hui, Shaolong Wu, Orestis Papakyriakopoulos, Alex Pentland, Nigel Collier, Jiaxin Pei
arXiv:2606. 10949v1 Announce Type: new Abstract: Persistent memory systems promise to make LLMs more helpful by storing user beliefs over time.
By Shelly Bensal, Axel Magnuson, Aparna Balagopalan, Daniel M. Bikel
arXiv:2610.25463v2 Announce Type: replace
Abstract: How well can transformer-based Large Language Models (LLMs) retain and recall memories in long contexts of dialogue? LLMs are known to forget long-...
By Owen Bianchi, Samer Aslan, Ryunosuke Saito
arXiv:2601. 22146v2 Announce Type: replace-cross Abstract: Due to limited supervised training data, large language models (LLMs) are typically pre-trained via a self-supervised "predict the next word" objective on a vast amount of unstructured text data.
By Ajay Patel, Colin Raffel, Chris Callison-Burch
arXiv:2609.15234v1 Announce Type: new
Abstract: Recently, Large Language Models (LLMs) have gained significant attention due to their strong language understanding and generation capabilities, demons...
By Keuntae Kim, Eunhye Jeong, Yong Suk Choi
arXiv:2607. 28657v1 Announce Type: new Abstract: Large Language Models (LLMs) often require carefully crafted prompts to unlock their full potential, which can be a barrier for non-expert users.
By Oliver Savolainen, Emanuele Bastianelli, Hosein Azarbonyad
arXiv:2610.22671v2 Announce Type: replace
Abstract: As assistants move from reactive answering toward proactive intervention, memory changes role: it no longer only supplies the answer to a query, it...
By Shenglai Zeng, Xiao Yang, Tiffany Ma, Xinyuan Zhang, Nanshu Wang, Deren Lei, Kai Sun, Xilun Chen, Tammy Stark, Xianhui Zhu, Yuetai Li, Rulin Shao, Tianxin Wei, Rakesh Wanga, Anuj Kumar, Jiliang Tang, Wen-tau Yih, Xin Luna Dong
arXiv:2605.06165v2 Announce Type: replace
Abstract: As the widespread adoption of Large Language Models (LLMs) accelerates, token consumption from intermediate reasoning traces increasingly contribut...
By Richmond Sin Jing Xuan, Rishabh Bhardwaj, Soujanya Poria
The paper introduces a method for determining whether retrieval-augmented generation (RAG) systems have sufficient, insufficient, or conflicting evidence to answer a question. By training a lightweight linear classifier on hidden activations and attention-derived features from 16 language models, the authors demonstrate that these internal signals reliably predict the adequacy of retrieved documents, outperforming prompting-based baselines and specialized RAG models. Analysis shows that middle-layer hidden states carry the most informative signals for this triage task.
By Syed Mahbubul Huq, Christopher Child, Tillman Weyde, Pranava Madhyastha
The paper introduces LRE (Learned Relevance Eviction), a lightweight, CPU‑only, language‑model‑free scorer that learns which parts of an agent’s interaction history are task‑critical and preserves them verbatim. In experiments, LRE matches or surpasses baseline eviction policies on accuracy‑cost trade‑offs, recovers 93% of full‑history accuracy, reduces worst‑case prompt size by 52%, and outperforms dense and token‑pruning encoders in conversational memory while being 295–1569× smaller. The method also achieves superior budgeted answer quality on LoCoMo reading and can be trained annotation‑free, recovering 95% of supervised scorer performance.
By Nusrat Jahan Lia, Aritra Mazumder