The Flow has not summarised this story yet — read it at Hugging Face Trending Papers.
The study evaluates how large language models (LLMs) interpret verbal probability expressions by mapping words to numbers and testing consistency across 19 models. Results show that LLMs largely mirror human benchmarks—preserving word order, recovering key anchor points, and reflecting the high variance of the term "possible"—but they exhibit a systematic upward bias for negative expressions like "unlikely" and "improbable." Explanation elicitation reduces within‑model variance but increases divergence between models, while a bidirectional roundtrip test reveals that leading models maintain coherent internal representations.
The paper examines how large language models (LLMs) express uncertainty compared to humans, noting that humans use verbal markers like "possible" or "likely" to convey metacognitive awareness. By curating a corpus of human uncertainty markers and benchmarking LLMs against it, the authors find that LLMs encode these markers with numerical levels that differ substantially from human usage. They introduce METHODNAME, an optimization-based algorithm that learns an optimal uncertainty profile over verbal markers directly from LLM outputs, enabling a direct comparison of confidence semantics and revealing systematic disparities in verbal expressions.
The paper investigates the relationship between a large language model’s internal probability distribution and its verbalized confidence statements. By systematically manipulating training and in‑context data, the authors show that both internal and verbalized probabilities are influenced by distributional and asserted uncertainty in the data. They find that verbalized probabilities align with internal ones beyond what would be expected if they tracked the same sources independently, indicating that verbalized confidence can serve as a probe of the model’s internal distribution.
As the chain-of-thought reasoning capabilities of large language models improve, evaluating and calibrating their reasoning confidence is becoming increasingly important for quantifying the uncertaint...
arXiv:2509.15114v2 Announce Type: replace Abstract: A controversial test for Large Language Models concerns the ability to discern possible from impossible language. While some evidence attests to th...
arXiv:2607. 03882v1 Announce Type: cross Abstract: LLMs are increasingly deployed as post-hoc explainers of AI-generated outputs, yet it remains unclear whether they can reliably communicate probabilistic information in natural language.