arXiv AI
Jul 14
Filtered Reasoning Score: Evaluating Reasoning Quality on a Model's Most-Confident Traces
arXiv:2604. 11996v2 Announce Type: replace-cross Abstract: Should we trust Large Language Models (LLMs) with high accuracy?
Can We Trust LLM's Logic? Quantifying Uncertainty, Coherence, and Robustness via a Graph-Based Framework
arXiv:2607. 08017v1 Announce Type: cross Abstract: Large-Language Models (LLMs) can be prone to flawed and unfaithful reasoning that decoding strategies like Self-Consistency (SC) fail to detect as they evaluate only final-answer agreement while ignoring the logical validity of intermediate steps.
Related stories
arXiv AI
Jul 21
Debate-on-Graph: Reliable and Adaptive Reasoning of Large Language Model on Uncertain Knowledge Graph
arXiv:2607. 17266v1 Announce Type: cross Abstract: Large language models (LLMs) have demonstrated remarkable capabilities in natural language processing.
arXiv AI
Jun 3
Quantifying Faithful Confidence Expression in Large Reasoning Models
arXiv:2606. 03969v1 Announce Type: cross Abstract: Reliable uncertainty communication is critical to the trustworthiness of LLMs, yet faithful calibration (FC)--the alignment between models' intrinsic and (linguistically) expressed confidence--is a persistent failure mode.
arXiv AI
Aug 11
Decodable but Not Faithful: Coupling Natural-Language Rationales to Programmatic Verifiers
arXiv:2606. 21678v2 Announce Type: replace-cross Abstract: Language models can generate plausible rationales for their predictions, but these explanations may not faithfully represent the model's internal reasoning.
arXiv AI
Jun 8
How Language Models Fail: Token-Level Signatures of Committed and Persistent Reasoning Failures
arXiv:2606. 06635v1 Announce Type: cross Abstract: Failures in language model reasoning emerge through distinct processes that leave identifiable signatures in the reasoning trace.
arXiv AI
Jun 17
Quantifying Consistency in LLM Logical Reasoning via Structural Uncertainty
arXiv:2606. 17312v1 Announce Type: new Abstract: Large language models can arrive at the same answer through reasoning paths that are unstable, contradictory, or difficult to rank consistently -- a failure mode especially prevalent in multi-step deductive reasoning.
arXiv AI
Jun 8
Stable Reasoning, Unstable Responses: Mitigating LLM Deception via Stability Asymmetry
arXiv:2603. 26846v2 Announce Type: replace-cross Abstract: As Large Language Models (LLMs) expand in capability and application scope, their trustworthiness becomes critical.
arXiv AI
Jul 1
Measuring Reasoning Quality in LLMs: A Multi-Dimensional Behavioral Framework
arXiv:2605. 24661v2 Announce Type: replace Abstract: LLMs have achieved remarkable success in complex reasoning tasks, yet current evaluation approaches predominantly rely on final-answer correctness, offering limited insight into the underlying reasoning processes that produce those answers.
arXiv AI
2d ago
The Metacognitive Bottleneck: Japanese Riddles Reveal Fundamental Limits of Machine Insight and Self-Evaluation in Reasoning AI
arXiv:2509. 14704v3 Announce Type: replace Abstract: Benchmark saturation and training-data contamination increasingly obscure whether reported gains in large language models (LLMs) reflect genuine advances in reasoning or familiarity with recurring patterns in benchmark problems.
arXiv AI
Jul 1
Disentangling Reasoning Logic to Resolve Explicit Knowledge Conflicts
arXiv:2508. 01273v3 Announce Type: replace Abstract: Explicit knowledge conflicts, occurring when retrieved contexts contain contradictory information, pose a fundamental challenge for Large Language Models (LLMs) as they integrate increasingly diverse data sources.
arXiv AI
Aug 5
The Tell-Tale Trace: Detecting Reasoning Failures in LLMs Using Chain-of-Thought Dynamics
arXiv:2608. 03291v1 Announce Type: cross Abstract: Chain-of-thought (CoT) reasoning improves large language model (LLM) performance while also providing an observable interface to the model's reasoning process.
arXiv AI
Jul 24
Post-Hoc Reasoning in Chain of Thought: Decoding and Steering Pre-Committed Answers
arXiv:2603. 01437v2 Announce Type: replace Abstract: As chain of thought (CoT) has become central to scaling reasoning capabilities in large language models (LLMs), it has also emerged as a promising tool for interpretability, suggesting the opportunity to understand model decisions through verbalized reasoning.