← Back to all news
arXiv AI By Christos Petridis, Konstantinos Pelechrinis, Zoran Obradovic

How Unlikely Is "Unlikely"? Assessing Verbal Probability Perception Across Large Language Models

Read the original on arXiv AI →

The study evaluates how large language models (LLMs) interpret verbal probability expressions by mapping words to numbers and testing consistency across 19 models. Results show that LLMs largely mirror human benchmarks—preserving word order, recovering key anchor points, and reflecting the high variance of the term "possible"—but they exhibit a systematic upward bias for negative expressions like "unlikely" and "improbable." Explanation elicitation reduces within‑model variance but increases divergence between models, while a bidirectional roundtrip test reveals that leading models maintain coherent internal representations.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

Related stories

arXiv Machine Learning
Oct 3

"very likely" Means "uncertain"? How LLMs Diverge from Humans in Linguistic Uncertainty Quantification

The paper examines how large language models (LLMs) express uncertainty compared to humans, noting that humans use verbal markers like "possible" or "likely" to convey metacognitive awareness. By curating a corpus of human uncertainty markers and benchmarking LLMs against it, the authors find that LLMs encode these markers with numerical levels that differ substantially from human usage. They introduce METHODNAME, an optimization-based algorithm that learns an optimal uncertainty profile over verbal markers directly from LLM outputs, enabling a direct comparison of confidence semantics and revealing systematic disparities in verbal expressions.

By Jinhao Duan, Zicheng Liu, Zijie Liu, Kaidi Xu, Tianlong Chen
llmsbenchmarks
More like this →
arXiv AI
Oct 2

Verbalized and Internal Probabilities Are Coupled in Large Language Models

The paper investigates the relationship between a large language model’s internal probability distribution and its verbalized confidence statements. By systematically manipulating training and in‑context data, the authors show that both internal and verbalized probabilities are influenced by distributional and asserted uncertainty in the data. They find that verbalized probabilities align with internal ones beyond what would be expected if they tracked the same sources independently, indicating that verbalized confidence can serve as a probe of the model’s internal distribution.

By Sinead Williamson, Jiaxuan Li, Nick Foti, Russ Webb, Masha Fedzechkina
llms
More like this →
arXiv Computation and Language
2d ago

Large Language Model probabilities cannot reliably distinguish between possible and impossible language

arXiv:2509.15114v2 Announce Type: replace Abstract: A controversial test for Large Language Models concerns the ability to discern possible from impossible language. While some evidence attests to th...

By Evelina Leivada, Raquel Montero, Paolo Morosi, Natalia Moskvina, Tamara Serrano, Nikoleta Pantelidou, Marcel Aguilar, Fritz Guenther
llmsbenchmarks
More like this →