arXiv Machine Learning
Does Reasoning Improve Psychological Depth in Large Language Models? It Depends on Who's Judging
Read the original on arXiv Machine Learning →The Flow has not summarised this story yet — read it at arXiv Machine Learning.
Related stories
arXiv Computation and Language
3d ago
When Consistency Does Not Mean Reliability: Evaluating Local LLM Judges Against Human Ratings
arXiv:2609.13824v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly used to evaluate the responses of other language models. This approach, known as LLM-as-a-Judge, is faste...
arXiv AI
Jun 9
SLMJury: Can Small Language Models Judge as Well as Large Ones?
arXiv:2606. 07810v1 Announce Type: cross Abstract: Large language models (LLMs) are widely used as judges for evaluating model outputs, but their high cost, latency, and opacity limit scalability.
arXiv Machine Learning
Jul 3
Ask the Right Comparison:Bias-Aware Bayesian Active Top-$k$ Ranking with LLM Judges
arXiv:2607. 02104v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly used as cheap, scalable judges that compare candidate outputs pairwise -- to rank responses, select models, or triage papers.
arXiv Computation and Language
Sep 2
Argument Quality Assessment with Large Language Models: A Pairwise Bradley-Terry Approach
arXiv:2605.28313v2 Announce Type: replace Abstract: Large Language Models (LLMs) have demonstrated remarkable capabilities in tasks related to reasoning and judgment. However, assessing the quality o...
arXiv Computation and Language
Sep 11
Rethinking Verbalized Confidence for LLM-as-a-Judge: A Compatibility Shift on Post-2025 Proprietary Models
The paper argues that verbalized confidence—once viewed as overconfident and coarse—has become the preferred soft‑scoring method for LLM‑as‑a‑Judge on top‑tier proprietary models released after 2025. Experiments on SummEval, AggreFact, and HelpSteer2 across up to 18 LLMs show that log‑probabilities are no longer the best signal, and that adding an overconfidence advisory and self‑debate further improves calibration and robustness. The authors note that these enhancements incur little accuracy loss on post‑2025 models but do affect pre‑2025 ones, highlighting a compatibility shift in how confidence should be measured.
arXiv AI
Jun 10
RealMath-Eval: Why SOTA Judges Struggle with Real Human Reasoning
arXiv:2606. 10254v1 Announce Type: new Abstract: While Large Language Models (LLMs) have achieved near-perfect performance in \emph{solving} high-school mathematics, their ability to \emph{evaluate} the diverse reasoning processes of real human students remains under-examined.