arXiv AI
Do Large Language Models Know What They Don't Know II? A Fully Behavioral, Non-Cognitive Measure of Epistemic Honesty
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv Computation and Language
Aug 25
Claim-Level Confidence Calibration for Reliable Decision Making with Large Language Models
arXiv:2608.22483v1 Announce Type: new Abstract: Large Language Models (LLMs) increasingly support decision-making in high-stakes domains, but they often hallucinate and express confidence that is mis...
arXiv AI
Jun 2
Asking Is Not Enough: Protocol Sensitivity in LLM Confidence Calibration
arXiv:2605. 27752v2 Announce Type: replace Abstract: LLM confidence calibration is often evaluated by comparing two signals: token-probability scores and verbalized confidence.
arXiv Machine Learning
Oct 7
Rethinking Faithfulness in LLMs: A Pairwise Context-Sensitive Perspective
arXiv:2610.07894v1 Announce Type: cross Abstract: Large language models (LLMs) are expected to answer questions faithfully based on the provided context, abstaining when the context information is in...
arXiv AI
Sep 25
PROOF: Profiling Reliability of Object-Level Facts in Large Language Models
PROOF is a benchmark that profiles the reliability of object-level facts in instruction-tuned language models by converting a frozen Wikidata snapshot into 18,486 English multiple-choice questions grounded in 11,779 semantic facts across 101 classes, 392 properties, and 14 domains. Each question includes an explicit "I don't know" option, a "No correct option" control, and nine controlled formulations, with 1,849 questions designed as no-correct-option traps. The study evaluates 18 open-weight model deployments on 166,374 prompts, revealing wide variability in factual accuracy, sensitivity to wording changes, and the impact of decoder perturbations.
arXiv AI
Aug 11
When Confidence Fails: Overconfidence in LLMs under Uncertainty and Missing Clinical Information
arXiv:2608. 09080v1 Announce Type: cross Abstract: Large Language Models (LLMs) have achieved strong performance in medical question answering and clinical reasoning tasks.
arXiv Machine Learning
Jul 9
Does Bielik Know What It Doesn't Know? Activation Dispersion Separates Entity Familiarity from Factual Reliability Across Model Scale
arXiv:2607. 07670v1 Announce Type: cross Abstract: Large language models hallucinate most about entities they have never seen.