← Back to all news
arXiv Computation and Language By Jaewon Sok, Jewon Yeom, Seonghyeon Park, Jeongjae Park, Hwiyeong Lee, Taesup Kim

On the Tip of the Tongue: First-Token Cueing as a Behavioral Probe of Recall Failures in LLMs

Read the original on arXiv Computation and Language →

The paper investigates why large language models often answer factual questions incorrectly in closed‑book settings. It shows that many errors are due to first‑token recall failures: the model commits to an incorrect first token but can complete the correct answer once the right token is provided. Using a First‑Token Cueing probe, the authors demonstrate that correcting just the first token recovers a significant portion of errors across 16 open models and improves accuracy on multilingual benchmarks.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computation and Language.

Related stories

arXiv AI
Oct 2

Certainty Is Not Just Correctness: Rethinking Token-Level Certainty in LLM Reasoning

The paper investigates token‑level certainty as a proxy for correctness in large language models. It finds that certainty better predicts whether a model will answer a question correctly than it does whether a specific response is correct, and that certainty varies by token type and position. The authors show that using certainty early in generation to allocate responses and later to weight votes improves accuracy while dramatically cutting token cost.

By Yunfan Zhou, Ye Zhu, Zhihai Wang, Jianguo Yao, Haibing Guan, Xijun Li
llms
More like this →
arXiv AI
Sep 30

Probability is Not Enough: Exploring and Counting Divergent Tokens for Reasoning Uncertainty Quantification in LLMs

The paper introduces Divergent Token Confidence (DTC), a method that estimates large language model confidence by counting tokens where two models strongly disagree during decoding. DTC uses Jensen-Shannon divergence between next-token distributions along the same reasoning trajectory and shows a near-negative correlation with answer accuracy. Experiments on multiple model families and six mathematical benchmarks demonstrate that DTC improves calibration over traditional probability-based and verbalized baselines, achieving lower expected calibration errors in both white-box and black-box settings.

By Feiyang Li, Shengjing Liu, Qi Zhan, Sijie Cheng, Weiqing Wang, Hongwen Chen, Yuxuan Yang, Wen Wang, Yile Wang, Hui Huang
llmsbenchmarks
More like this →