arXiv AI
Do Large Language Models Know What They Don't Know II? A Fully Behavioral, Non-Cognitive Measure of Epistemic Honesty
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv Computation and Language
Aug 25
Claim-Level Confidence Calibration for Reliable Decision Making with Large Language Models
arXiv:2608.22483v1 Announce Type: new Abstract: Large Language Models (LLMs) increasingly support decision-making in high-stakes domains, but they often hallucinate and express confidence that is mis...
arXiv AI
Jun 2
Asking Is Not Enough: Protocol Sensitivity in LLM Confidence Calibration
arXiv:2605. 27752v2 Announce Type: replace Abstract: LLM confidence calibration is often evaluated by comparing two signals: token-probability scores and verbalized confidence.
arXiv AI
Aug 11
When Confidence Fails: Overconfidence in LLMs under Uncertainty and Missing Clinical Information
arXiv:2608. 09080v1 Announce Type: cross Abstract: Large Language Models (LLMs) have achieved strong performance in medical question answering and clinical reasoning tasks.
arXiv Machine Learning
Jul 9
Does Bielik Know What It Doesn't Know? Activation Dispersion Separates Entity Familiarity from Factual Reliability Across Model Scale
arXiv:2607. 07670v1 Announce Type: cross Abstract: Large language models hallucinate most about entities they have never seen.
arXiv AI
Aug 28
Calibrated Enough to Know, Not Calibrated to Act: Fabricated Evidence Makes LLM Agents Commit to the Unknowable
The study shows that large language model (LLM) agents are far more likely to commit to a directional prediction when presented with a professional‑looking market panel than when asked the same question directly, with commitment rates rising from 6.5% to 54.0% across 12 frontier models. Even when the panel’s data is entirely fabricated, commitment still increases significantly, indicating that the authority of the presentation, rather than the truth of the information, drives confident action. The authors demonstrate that this act/don’t‑act decision gate is narrow, model‑specific, and can be mitigated through supervised fine‑tuning, though its effectiveness depends on response format and context. whyItMatters":"The findings reveal a specific vulnerability in LLMs where presentation style can override factual accuracy, highlighting the need for careful design and training to prevent misleading confidence in uncertain scenarios."
arXiv AI
Jul 23
Information Discernment in Large Language Models
arXiv:2607. 19355v1 Announce Type: new Abstract: LLMs are increasingly used with external knowledge sources like the internet.