Hugging Face Trending Papers

Locating Answer-Correctness Signals in Frozen Large Language Models

arXiv AI
Sep 30

Locating Answer-Correctness Signals in Frozen Large Language Models

The paper investigates where and how large language models encode signals that indicate answer correctness. By examining hidden states, token probabilities, residual-stream features, attention, and their combinations, the authors find that correctness signals are concentrated in the answer span and that different signal families complement each other. Fusing these signals improves robustness, especially under distribution shifts, and can be used to control retrieval in downstream tasks.

By Yuansen Liu, Yixuan Tang, Anthony Kum Hoe Tung
arXiv AI
Sep 3

Language Models Can Control Their Own Attention

The paper introduces Declarative Attention (DA), a protocol that lets language models explicitly declare which parts of their context to focus on during generation. By partitioning decoding into full-context, region-specific, and recent-output-only modes, the inference engine can skip large portions of the KV cache, dramatically reducing attended tokens. Experiments on 15 long-context tasks with off-the-shelf models show significant savings (52.0% and 31.1% reductions) with only modest accuracy drops that diminish as model size increases.

By Namgyu Ho, Huzama Ahmad, Woosung Koh, Se-Young Yun, Tal Schuster, Cicero Nogueira dos Santos
arXiv Computation and Language
Aug 31

Knowing Before Answering: Decoding Language Models for Reliable RAG

The paper introduces a method for determining whether retrieval-augmented generation (RAG) systems have sufficient, insufficient, or conflicting evidence to answer a question. By training a lightweight linear classifier on hidden activations and attention-derived features from 16 language models, the authors demonstrate that these internal signals reliably predict the adequacy of retrieved documents, outperforming prompting-based baselines and specialized RAG models. Analysis shows that middle-layer hidden states carry the most informative signals for this triage task.

By Syed Mahbubul Huq, Christopher Child, Tillman Weyde, Pranava Madhyastha
arXiv Computation and Language
Oct 19

On the Tip of the Tongue: First-Token Cueing as a Behavioral Probe of Recall Failures in LLMs

The paper investigates why large language models often answer factual questions incorrectly in closed‑book settings. It shows that many errors are due to first‑token recall failures: the model commits to an incorrect first token but can complete the correct answer once the right token is provided. Using a First‑Token Cueing probe, the authors demonstrate that correcting just the first token recovers a significant portion of errors across 16 open models and improves accuracy on multilingual benchmarks.

By Jaewon Sok, Jewon Yeom, Seonghyeon Park, Jeongjae Park, Hwiyeong Lee, Taesup Kim
arXiv AI
Nov 6

Circe: Training-Free LLM Self-Evaluation via Component Interaction Consistency

Circe is a training‑free method for self‑evaluating large language model responses without reference answers. It combines internal signals—component interaction consistency (CIC) and representation geometry (RG)—with an external likelihood confidence (LC) score to produce a unified correctness score. Experiments on five LLMs across six benchmarks demonstrate that Circe better distinguishes correct from incorrect responses than existing baselines and works in multilingual, multimodal, and open‑world settings.

By Shuyi Miao, Zheng Li, Xu Xie, Linrui Xu, Fei Shen, Xiaoyu Du, Tat-Seng Chua
arXiv Machine Learning
Oct 23

Mixing Matters: Evidence Position Bias Across Sequence Mixers in Long-Context Question Answering

The study investigates how the position of evidence documents relative to a prompt affects long‑context question answering across different sequence mixers—Transformer, state‑space, and hybrid Mamba models. Results show that Transformers exhibit a beginning‑position advantage that state‑space models lack, while both families tend to favor evidence near the end. Additional experiments reveal that prompt design, corpus choice, and model capability influence performance, and that position bias remains decodable across architectures even when response curves differ.

By Aman Behera, Namit Solanki, Mehul Anand
arXiv AI
Sep 18

How to Guide Your Language Flow

The paper introduces probe guidance, a technique that leverages frozen internal states of a diffusion model to generate a guidance signal without requiring an extra forward pass during inference. This method improves continuous diffusion language models, achieving state‑of‑the‑art results on unconditional generation and enhancing performance on multiple‑choice question answering for a 1.7B model. The authors also use probes to analyze autoguidance, revealing that the weak model must originate from a low‑entropy training region to align dynamics with the strong model.

By Rohit Dilip, Tianrong Chen, Yuyang Wang, David Van Valen, Joshua Susskind, Miguel Angel Bautista
arXiv Computation and Language
Sep 15

Where to Look and What to Use: Retrieve-Localize-Generate for Long-Term Conversational Memory Question Answering

arXiv:2609.07093v2 Announce Type: replace Abstract: Retrieval-augmented generation (RAG) enables large language models (LLMs) to answer questions by accessing external knowledge and has been widely a...

By Yifan Wang, Xinkui Lin, Yongxiu Xu, Shen Gao, Ruochen Yang, Kun Huang, Yubin Wang, Jie Wu, Wei Liu, Jian Luan, Hongbo Xu, Shuo Shang
arXiv AI
Oct 1

NinaXander: Feasibility and Limits of Composing Frozen Language Models Across Architecture Families via a Shared Latent Space

The paper introduces NinaXander, a method for composing frozen language models from different architecture families by inserting a trained shared‑latent adapter between their layers. By running the initial layers of one model, converting the intermediate representation with the adapter, and then continuing with the remaining layers of another model, multiple composed models can be created without retraining. Experiments with RWKV and Pythia show that while some compositions preserve syntactic quality and reduce memory usage, none match the parent model’s accuracy and language‑modeling performance drops on out‑of‑domain data.

By Takanori Kotama, Shun-ichiro Hayashi, Daichi Mukunoki, Tetsuya Hoshino, Takahiro Katagiri