The paper introduces CoDIT, a contrastive decoding technique that separates instruction-following behavior from pre-trained world knowledge in large language models. By generating responses that emphasize post-training instruction capabilities while suppressing shared pre-trained knowledge, CoDIT creates instruction-tuning datasets that lead to consistently better model performance than directly generated responses or existing public datasets. The authors also provide theoretical and empirical evidence that CoDIT effectively distills instruction-tuning knowledge from model parameters into text, facilitating cross-architecture transfer.
By Tatsuya Ichinose, Youmi Ma, Masanari Oi, Ryuto Koike, Naoaki Okazaki
arXiv:2606. 05644v1 Announce Type: new Abstract: When retrieved evidence contradicts parametric memory, language models frequently ignore context and default to memorized priors -- a failure that undermines the core purpose of retrieval augmentation.
By Zhe Yu, Wenpeng Xing, Tiancheng Zhao, Mohan Li, Changting Lin, Meng Han
arXiv:2607. 04223v1 Announce Type: cross Abstract: Retrieval-augmented generation (RAG) reduces but does not eliminate hallucination, and existing detectors return a single answer-level score that does not indicate which sentence is unsupported, or why.
By Mohamed Aly Bouke
arXiv:2607. 20379v1 Announce Type: new Abstract: Natural-language autoencoders score explanations of hidden activations by reconstruction: an explanation is deemed faithful if the activation can be regenerated from it.
By Hiskias Dingeto
arXiv:2607. 19257v1 Announce Type: cross Abstract: Practitioners make three prompt-design decisions with almost no controlled evidence behind them: how to format instructions and context (markdown, plain text, prose, or tabular), how many simultaneous instructions a system prompt can carry before compliance degrades, and how much context a model can hold before recall and honesty degrade.
By Netanel Eliav
arXiv:2607. 03502v1 Announce Type: cross Abstract: Frontier LLMs can perform multi-step reasoning over content-free filler tokens like dots or counting sequences, producing correct answers with no visible chain-of-thought (CoT).
By Kaley Brauer, Claudio Mayrink Verdun, Samuel Marks
arXiv:2605.24614v2 Announce Type: replace-cross
Abstract: Large language model (LLM) unlearning has emerged as a crucial post-hoc mechanism for privacy protection and AI safety, yet auditing whether...
By Jaeung Lee, Dohyun Kim, Jaemin Jo
LLM-based evaluators of natural language generation (NLG) quality are widely deployed as scoring tools and as automated training signals, yet the internal procedure by which they assign a rating remai...
arXiv:2605. 28969v2 Announce Type: replace-cross Abstract: If an AI agent makes decisions on a person's behalf, those decisions must align with its user.
By Aarik Gulaya
arXiv:2609.01604v1 Announce Type: cross
Abstract: LLM-based evaluators of natural language generation (NLG) quality are widely deployed as scoring tools and as automated training signals, yet the int...
By Himil Vasava, Ming Jiang
The paper introduces Exemplar Partitioning (EP), an unsupervised technique that constructs interpretable feature dictionaries from large language model activations by clustering streamed activations into Voronoi regions defined by exemplars and their averages. EP allows comparison of dictionaries across layers, checkpoints, and architectures, and demonstrates utility in interpreting model behavior, tracking training dynamics, detecting hidden concepts, and enabling targeted interventions. Experiments on Gemma‑2‑2B and Llama‑3.1‑8B show EP can reveal how instruction tuning reorganizes harmful prompt activations, facilitate interventions that alter model responses, and achieve high concept‑detection performance while requiring far fewer construction tokens than comparable methods.
By Jessica Rumbelow
arXiv:2602.00377v3 Announce Type: replace
Abstract: Existing knowledge probing methods rely on pre-defined queries, limiting extraction to known concepts. We introduce DecompressionLM, a stateless fr...
By Zhaochen Hong, Jiaxuan You