Judge Arena: Benchmarking LLMs as Evaluators

Understanding the 4 Main Approaches to LLM Evaluation (From Scratch)

Multiple-Choice Benchmarks, Verifiers, Leaderboards, and LLM Judges with Code Examples

When the Judge Changes, So Does the Measurement: Auditing LLM-as-Judge Reliability

arXiv:2607. 08535v1 Announce Type: cross Abstract: An LLM-as-judge score can move even when the candidate responses stay fixed, simply because the evaluator has changed.

Quantifying and Mitigating Self-Preference Bias of LLM Judges

arXiv:2604. 22891v4 Announce Type: replace-cross Abstract: LLM-as-a-Judge has become a dominant approach in automated evaluation systems, playing critical roles in model alignment, leaderboard construction, quality control, and so on.

Rethinking LLM Evaluation with 3C3H: AraGen Benchmark and Leaderboard

"LLM Agent Performance" Is Not a Single Evaluation Target

arXiv:2602. 03238v3 Announce Type: replace Abstract: LLM agent benchmark scores are shaped not only by the model but also by the agent harness, environment, evaluator, and inference budget.

Eta Given Delta: Defining LLM Tool Efficiency With Marginal Tool Utility

arXiv:2607. 14108v1 Announce Type: cross Abstract: This paper introduces tool efficiency, a new quantitative metric to evaluate the rate of useful tool calls in an LLM agent trajectory.

PoQ-Judge: A Multi-Architecture Evaluation Framework for Cost-Aware Proof-of-Quality in Decentralized LLM Inference

arXiv:2606. 11196v1 Announce Type: cross Abstract: Decentralized LLM inference networks need lightweight, reference-free quality evaluation for Proof of Quality (PoQ).

Benchmarking LLM Judges for Mobile Agent Evaluation

arXiv:2608. 11434v1 Announce Type: new Abstract: Mobile agent benchmarks increasingly rely on LLM-based judges to evaluate task completion, yet the reliability of these judges on mobile agent trajectories remains largely unexamined.

Introducing the Open Ko-LLM Leaderboard: Leading the Korean LLM Evaluation Ecosystem

Evaluating RAG with LLM as a Judge

Cost-Effective Automated Judging of Natural-Language Mathematical Proofs

arXiv:2608. 00004v1 Announce Type: cross Abstract: Grading natural-language mathematical proofs is a recurring cost in evaluating math-reasoning systems, and frontier LLM judges are expensive.

Auditing Automated Evaluation, Error Propagation, and Runtime Mitigation in Tool-Using Language Agents

arXiv:2604. 16706v2 Announce Type: replace Abstract: Automated evaluation of tool-using large language model (LLM) agents is widely assumed to be reliable, yet this assumption is rarely validated against human annotation.