The paper investigates where and how large language models encode signals that indicate answer correctness. By examining hidden states, token probabilities, residual-stream features, attention, and their combinations, the authors find that correctness signals are concentrated in the answer span and that different signal families complement each other. Fusing these signals improves robustness, especially under distribution shifts, and can be used to control retrieval in downstream tasks.
By Yuansen Liu, Yixuan Tang, Anthony Kum Hoe Tung
The paper introduces Declarative Attention (DA), a protocol that lets language models explicitly declare which parts of their context to focus on during generation. By partitioning decoding into full-context, region-specific, and recent-output-only modes, the inference engine can skip large portions of the KV cache, dramatically reducing attended tokens. Experiments on 15 long-context tasks with off-the-shelf models show significant savings (52.0% and 31.1% reductions) with only modest accuracy drops that diminish as model size increases.
By Namgyu Ho, Huzama Ahmad, Woosung Koh, Se-Young Yun, Tal Schuster, Cicero Nogueira dos Santos
The paper introduces a method for determining whether retrieval-augmented generation (RAG) systems have sufficient, insufficient, or conflicting evidence to answer a question. By training a lightweight linear classifier on hidden activations and attention-derived features from 16 language models, the authors demonstrate that these internal signals reliably predict the adequacy of retrieved documents, outperforming prompting-based baselines and specialized RAG models. Analysis shows that middle-layer hidden states carry the most informative signals for this triage task.
By Syed Mahbubul Huq, Christopher Child, Tillman Weyde, Pranava Madhyastha
The paper investigates why large language models often answer factual questions incorrectly in closed‑book settings. It shows that many errors are due to first‑token recall failures: the model commits to an incorrect first token but can complete the correct answer once the right token is provided. Using a First‑Token Cueing probe, the authors demonstrate that correcting just the first token recovers a significant portion of errors across 16 open models and improves accuracy on multilingual benchmarks.
By Jaewon Sok, Jewon Yeom, Seonghyeon Park, Jeongjae Park, Hwiyeong Lee, Taesup Kim
Circe is a training‑free method for self‑evaluating large language model responses without reference answers. It combines internal signals—component interaction consistency (CIC) and representation geometry (RG)—with an external likelihood confidence (LC) score to produce a unified correctness score. Experiments on five LLMs across six benchmarks demonstrate that Circe better distinguishes correct from incorrect responses than existing baselines and works in multilingual, multimodal, and open‑world settings.
By Shuyi Miao, Zheng Li, Xu Xie, Linrui Xu, Fei Shen, Xiaoyu Du, Tat-Seng Chua
arXiv:2609.37491v1 Announce Type: cross
Abstract: Retrieval-augmented language models are expected to answer from the retrieved evidence, but in practice they often keep answering when that evidence...
By Zeyan Li, Qirong Guo, SIyuan Qiu, Hu Xu, Chun Li, Jianfeng Xu
arXiv:2606. 27359v1 Announce Type: cross Abstract: Many decoding methods for large language models can be understood as shifting probability mass toward outputs that are more likely under the model, either locally at the token level or globally at the sequence level.
By Johannes Zenn, Jonas Geiping
The study investigates how the position of evidence documents relative to a prompt affects long‑context question answering across different sequence mixers—Transformer, state‑space, and hybrid Mamba models. Results show that Transformers exhibit a beginning‑position advantage that state‑space models lack, while both families tend to favor evidence near the end. Additional experiments reveal that prompt design, corpus choice, and model capability influence performance, and that position bias remains decodable across architectures even when response curves differ.
By Aman Behera, Namit Solanki, Mehul Anand
The paper introduces probe guidance, a technique that leverages frozen internal states of a diffusion model to generate a guidance signal without requiring an extra forward pass during inference. This method improves continuous diffusion language models, achieving state‑of‑the‑art results on unconditional generation and enhancing performance on multiple‑choice question answering for a 1.7B model. The authors also use probes to analyze autoguidance, revealing that the weak model must originate from a low‑entropy training region to align dynamics with the strong model.
By Rohit Dilip, Tianrong Chen, Yuyang Wang, David Van Valen, Joshua Susskind, Miguel Angel Bautista
arXiv:2606. 27679v1 Announce Type: cross Abstract: Probe-based uncertainty estimation (UE) has emerged as a prominent approach to detect hallucinations in Large Language Models (LLMs) by learning uncertainty from internal model signals.
By Ponhvoan Srey, Xiaobao Wu, Cong-Duy Nguyen, Quang Minh Nguyen, Duc Anh Vu, Anh Tuan Luu
arXiv:2609.07093v2 Announce Type: replace
Abstract: Retrieval-augmented generation (RAG) enables large language models (LLMs) to answer questions by accessing external knowledge and has been widely a...
By Yifan Wang, Xinkui Lin, Yongxiu Xu, Shen Gao, Ruochen Yang, Kun Huang, Yubin Wang, Jie Wu, Wei Liu, Jian Luan, Hongbo Xu, Shuo Shang
The paper introduces NinaXander, a method for composing frozen language models from different architecture families by inserting a trained shared‑latent adapter between their layers. By running the initial layers of one model, converting the intermediate representation with the adapter, and then continuing with the remaining layers of another model, multiple composed models can be created without retraining. Experiments with RWKV and Pythia show that while some compositions preserve syntactic quality and reduce memory usage, none match the parent model’s accuracy and language‑modeling performance drops on out‑of‑domain data.
By Takanori Kotama, Shun-ichiro Hayashi, Daichi Mukunoki, Tetsuya Hoshino, Takahiro Katagiri