arXiv AI July 15, 2026 By Jon-Paul Cacioli

Do LLMs Know What They Know? Measuring Metacognitive Efficiency with Signal Detection Theory

arXiv:2603. 25112v2 Announce Type: replace-cross Abstract: Standard evaluation of LLM confidence relies on calibration metrics (ECE, Brier score) that conflate how much a model knows (Type-1 accuracy) with how well its confidence signal tracks that knowledge (Type-2 metacognitive sensitivity).

Summary generated by The Flow from the publisher's feed. The full article lives at arXiv AI.

llms

arXiv AI

Aug 3

metasignal: A Python Package for Comprehensive Metacognitive Analysis and Decision-Making

arXiv:2607. 29093v1 Announce Type: cross Abstract: Metasignal is an open-source Python package for signal detection theory (SDT) and metacognitive measurement.

By Saurabh Ranjan, Mukesh Makwana, Konstantina Sokratous, Brian Odegaard

safety

arXiv AI

Jul 1

Reinforcement Learning with Metacognitive Feedback Elicits Faithful Uncertainty Expression in LLMs

arXiv:2606. 32032v1 Announce Type: cross Abstract: Metacognition is a critical component of intelligence that describes the ability to monitor and regulate one's own cognitive processes.

By Gabrielle Kaili-May Liu, Avi Caciularu, Gal Yona, Idan Szpektor, Arman Cohan

llms reinforcement-learning benchmarks safety

arXiv AI

1d ago

Large Language Models Show Metacognitive Sensitivity in Medical Reasoning

arXiv:2608. 14552v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly evaluated and used in medicine, but clinical usefulness depends on answer accuracy and whether confidence tracks evidence quality and uncertainty.

By Ahmad Nazzal

llms benchmarks

arXiv AI

Jun 19

LLM Doesn't Know What It Doesn't Know: Detecting Epistemic Blind Spots via Cross-Model Attribution Divergence on Clinical Tabular Data

arXiv:2606. 19509v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly applied to structured clinical data, yet whether they can recognize the limits of their own knowledge on such tasks remains unexplored.

By Akshat Dasula, Prasanna Desikan, Jaideep Srivastava

llms

arXiv AI

Jun 30

An Integrated Machine Learning and Hierarchical Variance Decomposition Pipeline for Student Performance Prediction and Metacognitive Calibration on Multi-Signal Telemetry

arXiv:2606. 28881v1 Announce Type: cross Abstract: Predicting student performance and characterizing metacognitive calibration are essential for personalization in intelligent tutoring systems.

By Gurdeep Singh Virdee

safety

arXiv AI

Jul 9

Measuring the metacognition of AI

arXiv:2603. 29693v3 Announce Type: replace Abstract: A robust decision-making process must take into account uncertainty, especially when the choice involves inherent risks.

By Richard Servajean, Philippe Servajean

llms

Related stories

metasignal: A Python Package for Comprehensive Metacognitive Analysis and Decision-Making

Reinforcement Learning with Metacognitive Feedback Elicits Faithful Uncertainty Expression in LLMs

Large Language Models Show Metacognitive Sensitivity in Medical Reasoning

LLM Doesn't Know What It Doesn't Know: Detecting Epistemic Blind Spots via Cross-Model Attribution Divergence on Clinical Tabular Data

An Integrated Machine Learning and Hierarchical Variance Decomposition Pipeline for Student Performance Prediction and Metacognitive Calibration on Multi-Signal Telemetry

Measuring the metacognition of AI