arXiv AI July 15, 2026 By Jon-Paul Cacioli

Do LLMs Know What They Know? Measuring Metacognitive Efficiency with Signal Detection Theory

arXiv:2603. 25112v2 Announce Type: replace-cross Abstract: Standard evaluation of LLM confidence relies on calibration metrics (ECE, Brier score) that conflate how much a model knows (Type-1 accuracy) with how well its confidence signal tracks that knowledge (Type-2 metacognitive sensitivity).

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

llms

arXiv AI

Aug 3

metasignal: A Python Package for Comprehensive Metacognitive Analysis and Decision-Making

arXiv:2607. 29093v1 Announce Type: cross Abstract: Metasignal is an open-source Python package for signal detection theory (SDT) and metacognitive measurement.

By Saurabh Ranjan, Mukesh Makwana, Konstantina Sokratous, Brian Odegaard

safety

arXiv AI

Jul 1

Reinforcement Learning with Metacognitive Feedback Elicits Faithful Uncertainty Expression in LLMs

arXiv:2606. 32032v1 Announce Type: cross Abstract: Metacognition is a critical component of intelligence that describes the ability to monitor and regulate one's own cognitive processes.

By Gabrielle Kaili-May Liu, Avi Caciularu, Gal Yona, Idan Szpektor, Arman Cohan

llms reinforcement-learning benchmarks safety

arXiv AI

Sep 21

How do LLMs Compute Verbal Confidence

arXiv:2603.17839v4 Announce Type: replace-cross Abstract: Verbal confidence -- prompting LLMs to state their confidence as a number or category -- is widely used to extract uncertainty estimates from...

By Dharshan Kumaran, Arthur Conmy, Federico Barbero, Simon Osindero, Viorica Patraucean, Petar Veli\v{c}kovi\'c

llms benchmarks

arXiv AI

Aug 18

Large Language Models Show Metacognitive Sensitivity in Medical Reasoning

arXiv:2608. 14552v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly evaluated and used in medicine, but clinical usefulness depends on answer accuracy and whether confidence tracks evidence quality and uncertainty.

By Ahmad Nazzal

llms benchmarks

arXiv Machine Learning

Sep 11

Evidence for Limited Metacognition in LLMs

The paper introduces a new method for measuring metacognitive abilities in large language models (LLMs) without relying on self-reports, instead testing how well models can use knowledge of their internal states. Using two experimental paradigms, the authors find that recent frontier LLMs can assess and use their own confidence when answering factual and reasoning questions, and can anticipate and appropriately employ the answers they would give. The study also shows that these abilities are limited in resolution, context-dependent, differ qualitatively from human metacognition, and vary across models with similar capabilities, suggesting post‑training processes influence metacognitive development.

By Christopher Ackerman

llms safety

arXiv AI

Jun 19

LLM Doesn't Know What It Doesn't Know: Detecting Epistemic Blind Spots via Cross-Model Attribution Divergence on Clinical Tabular Data

arXiv:2606. 19509v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly applied to structured clinical data, yet whether they can recognize the limits of their own knowledge on such tasks remains unexplored.

By Akshat Dasula, Prasanna Desikan, Jaideep Srivastava

llms

One email a morning, machine-written

Related stories

metasignal: A Python Package for Comprehensive Metacognitive Analysis and Decision-Making

Reinforcement Learning with Metacognitive Feedback Elicits Faithful Uncertainty Expression in LLMs

How do LLMs Compute Verbal Confidence

Large Language Models Show Metacognitive Sensitivity in Medical Reasoning

Evidence for Limited Metacognition in LLMs

LLM Doesn't Know What It Doesn't Know: Detecting Epistemic Blind Spots via Cross-Model Attribution Divergence on Clinical Tabular Data