arXiv AI August 3, 2026 By Nayera Hasan, Jack Greff, Alvin Grissom II

Benchmarking LLM Competence on Logical Inference over Probability Operators

arXiv:2607. 27405v2 Announce Type: replace-cross Abstract: Both expressions of uncertainty and inferences are ubiquitous in natural language, and valid inferences over natural-language expressions of uncertainty are necessary for not only everyday conversations but also for high-stakes domains such as medicine and law.

Summary generated by The Flow from the publisher's feed. The full article lives at arXiv AI.

arXiv AI

Jul 8

Estimating Uncertainty from Reasoning: A Large-Scale Study of Multi- and Crosslingual MCQA Performance in LLMs

arXiv:2607. 06327v1 Announce Type: cross Abstract: Uncertainty estimation (UE) enables LLM-powered systems to recognize when to abstain, yet existing research has predominantly focused on English.

By Andrea Alfarano, Andrea Bacciu, Saab Mansour, Amin Mantrach, Marcello Federico

llms rag

arXiv AI

Jun 8

How reliable are LLMs when it comes to playing dice?

arXiv:2606. 07515v1 Announce Type: cross Abstract: We investigate the probabilistic reasoning capabilities of large language models through a controlled benchmarking study on discrete probability problems.

By Luca Avena, Gianmarco Bet, Bernardo Busoni

llms rag benchmarks safety

arXiv AI

Jul 17

Simplicity Paradox: Debunking myths about prompting and datasets for LLM evaluation

arXiv:2607. 14109v1 Announce Type: cross Abstract: Probing the capabilities of Large Language Models (LLMs) and building robust solutions for Multiple-Choice Question Answering (MCQA) remain central challenges in natural language understanding.

By Inder Preet, Shuxin Lin, Dhaval Patel

llms nlp benchmarks

arXiv AI

Jul 21

Beyond Semantic Equivalence: Logical Graphs for LLM Uncertainty Quantification

arXiv:2607. 16868v1 Announce Type: new Abstract: Large Language Models (LLMs) often produce confidently stated yet unreliable outputs, posing critical challenges for deployment in safety-sensitive applications.

By Yanni Dong, Minghua Liu, Meiling Zhu, Xiaowei Huang, Lijun Zhang

llms benchmarks safety

arXiv AI

Jun 24

Benchmarking LLMs' Mathematical Reasoning with Unseen Random Variables Questions

arXiv:2501. 11790v5 Announce Type: replace-cross Abstract: Recent studies have raised significant concerns regarding the reliability of current mathematics benchmarks, highlighting issues such as simplistic design and potential data contamination.

By Zijin Hong, Hao Wu, Su Dong, Junnan Dong, Yilin Xiao, Yujing Zhang, Zhu Wang, Feiran Huang, Linyi Li, Hongxia Yang, Xiao Huang

llms benchmarks

arXiv AI

Jul 28

Same Question, Different Answers: Evaluating LLM Reliability Beyond Accuracy

arXiv:2607. 22554v1 Announce Type: new Abstract: Large language models (LLMs) often achieve strong accuracy on benchmarks, yet it remains unclear how reliably they apply this knowledge when the same question is phrased in different but equivalent ways.

By Kazem Faghih, Yize Cheng, Shoumik Saha, Mobina Pournemat, Armin Gerami, Soheil Feizi

llms nlp benchmarks

Related stories

Estimating Uncertainty from Reasoning: A Large-Scale Study of Multi- and Crosslingual MCQA Performance in LLMs

How reliable are LLMs when it comes to playing dice?

Simplicity Paradox: Debunking myths about prompting and datasets for LLM evaluation

Beyond Semantic Equivalence: Logical Graphs for LLM Uncertainty Quantification

Benchmarking LLMs' Mathematical Reasoning with Unseen Random Variables Questions

Same Question, Different Answers: Evaluating LLM Reliability Beyond Accuracy