arXiv AI August 14, 2026 By Congchao Wang, Diwakar Singh, Qiaozi Gao, Spyros Matsoukas, Yang Liu, Mahdi Namazifar

Reasoning Jury: Multi-Model Consensus for Evaluating Reasoning Traces

arXiv:2608. 12585v1 Announce Type: new Abstract: Improving reasoning LLMs requires the ability to judge the quality of long reasoning traces for effective reasoning data curation, strong training signals during reinforcement learning, and an in-depth understanding of reasoning behaviors during model performance evaluation.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

arXiv AI

Jul 14

Filtered Reasoning Score: Evaluating Reasoning Quality on a Model's Most-Confident Traces

arXiv:2604. 11996v2 Announce Type: replace-cross Abstract: Should we trust Large Language Models (LLMs) with high accuracy?

By Manas Pathak, Xingyao Chen, Shuozhe Li, Amy Zhang, Liu Leqi

llms benchmarks

arXiv AI

Jun 15

C2-Faith: Benchmarking LLM Judges for Causal and Coverage Faithfulness in Chain-of-Thought Reasoning

arXiv:2603. 05167v2 Announce Type: replace-cross Abstract: Large language models (LLMs) are increasingly used as judges of chain-of-thought (CoT) reasoning, yet it remains unclear whether they can reliably assess process faithfulness rather than merely answer plausibility.

By Avni Mittal, Rauno Arike

llms benchmarks

arXiv AI

Jul 1

Modality-Driven Search with Holistic Trace Judging for ARC-AGI-2

arXiv:2606. 31543v1 Announce Type: new Abstract: Large language models can produce fluent, internally coherent reasoning traces for abstract reasoning tasks while still being confidently wrong - making selection among candidates, not just generation, the central challenge.

By Johan Land

llms benchmarks

arXiv AI

Jul 14

LLMs as a Jury: Cross-Model Consensus Can Outperform Process Reward Models for LLM Reasoning

arXiv:2607. 10139v1 Announce Type: cross Abstract: Selecting the correct answer from a pool of candidate reasoning chains is the engine of test-time scaling, yet the standard selectors each carry a cost: self-consistency inherits the errors of the single model it resamples, and trained reward models need labeled data and transfer poorly off-distribution.

By Ning Liu

llms benchmarks

arXiv AI

Sep 1

JudgePanel: A Compact Judge with Panel Deliberation via Adaptive Multi-Reward Reinforcement Learning

arXiv:2608.29168v1 Announce Type: new Abstract: The LLM-as-a-Judge paradigm has emerged as a scalable alternative to human evaluation. However, single-model judges are limited by their inherent model...

By Yiyue Qian, Shinan Zhang, Huan Song, Hannah Marlowe

llms agents reinforcement-learning fine-tuning benchmarks

arXiv AI

3d ago

ArgGYM: A Procedural, Engine-Verified Benchmark for Structured Defeasible Reasoning

arXiv:2609.38409v1 Announce Type: new Abstract: Recent progress in large language model reasoning has been driven by benchmarks and reinforcement learning environments with automatically verifiable r...

By \.Ibrahim Ethem Deveci, Funda Tan \c{C}al{\i}k, Bar{\i}\c{s} Deniz Sa\u{g}lam, Duygu Ataman

llms reinforcement-learning benchmarks

One email a morning, machine-written

Related stories

Filtered Reasoning Score: Evaluating Reasoning Quality on a Model's Most-Confident Traces

C2-Faith: Benchmarking LLM Judges for Causal and Coverage Faithfulness in Chain-of-Thought Reasoning

Modality-Driven Search with Holistic Trace Judging for ARC-AGI-2

LLMs as a Jury: Cross-Model Consensus Can Outperform Process Reward Models for LLM Reasoning

JudgePanel: A Compact Judge with Panel Deliberation via Adaptive Multi-Reward Reinforcement Learning

ArgGYM: A Procedural, Engine-Verified Benchmark for Structured Defeasible Reasoning