arXiv AI August 20, 2026 By Chenle Chen, Yangbo Wei, Chao Yao, Shaoqiang Lu, Junhong Qian, Chen Wu, Lei He

Competence, Not Accuracy: A Diagnostic for Reference-Free Judge Gates in Skill Optimization

The paper introduces a diagnostic for reference‑free judge gates in text‑space skill optimization. It formalizes a judge as a latent solver, deriving a closed‑form bound on discriminability (ROC‑AUC) in terms of judge competence and answer‑space size, and shows that discriminability is confounded by item difficulty unless a within‑question estimator is used. A non‑intervening probe demonstrates that discriminability is at chance near the competence floor, rises above it, and that the diagnostic can predict gating errors in closed‑loop experiments.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

arXiv Machine Learning

Jul 8

More Convincing, Not More Correct: Self-Play Reward Hacking of Reference-Free LLM Judges

arXiv:2607. 05904v1 Announce Type: new Abstract: Training a language model against its own reference-free judgments (the premise of self-rewarding, self-play, and LLM-as-a-judge pipelines) assumes a model's verdict on a shown answer tracks correctness.

By Chenyu Zhou

llms reinforcement-learning

arXiv AI

Aug 18

Admission Without Answers: Label-Free Certification and Experience Learning for LLM-Based Optimization Modeling

arXiv:2608. 15565v1 Announce Type: new Abstract: Experience-learning agents for optimization modeling improve by storing verified skills, but existing learners admit knowledge by checking against known answers, which real ticket streams do not provide.

By Junbo Jacob Lian, Huiling Chen, Hanzhang Qin, Chung-Piaw Teo

llms agents benchmarks

Hugging Face Trending Papers

Jul 30

Share the Judge, Learn the Deferral: Where Specialization Helps LLM Evaluation

Agentic systems have widened the gap between producing candidate outputs and reviewing them. This paper asks a practical architectural question: should domain specialization be built into an evaluator's weights, or into the rule that decides when its judgment can be trusted?

llms agents fine-tuning

arXiv AI

Jul 14

LLMs as a Jury: Cross-Model Consensus Can Outperform Process Reward Models for LLM Reasoning

arXiv:2607. 10139v1 Announce Type: cross Abstract: Selecting the correct answer from a pool of candidate reasoning chains is the engine of test-time scaling, yet the standard selectors each carry a cost: self-consistency inherits the errors of the single model it resamples, and trained reward models need labeled data and transfer poorly off-distribution.

By Ning Liu

llms benchmarks

Hugging Face Trending Papers

Aug 18

Judge, Retrieve, or Abstain: Uncertainty-Guarded LLM Judging with Provable Risk Guarantees

The paper introduces a risk‑controlled framework for using large language models (LLMs) as judges in tasks without reference answers. By calibrating uncertainty thresholds on a held‑out set, the method ensures that the false discovery rate of accepted verdicts stays below a user‑specified level α with high probability, using finite‑sample Clopper–Pearson intervals. When the parametric judge lacks confidence, the instance is routed to a retrieval‑augmented mode with a second calibrated threshold, preserving the error guarantee while achieving higher coverage than single‑mode baselines.

llms rag benchmarks safety

arXiv AI

Aug 11

When the Judge Should Not Decide: Evidence-Locked, Non-Compensatory Selection Bounds LLM-Judge Failure in Reasoning Pipelines

arXiv:2608. 07813v1 Announce Type: new Abstract: An LLM judge deployed inside a reasoning pipeline does not merely measure quality, it decides which answer ships.

By Yiyao Zhang, Diksha Goel, Hussain Ahmad, Shixun Huang, Jun Shen

llms

One email a morning, machine-written

Related stories

More Convincing, Not More Correct: Self-Play Reward Hacking of Reference-Free LLM Judges

Admission Without Answers: Label-Free Certification and Experience Learning for LLM-Based Optimization Modeling

Share the Judge, Learn the Deferral: Where Specialization Helps LLM Evaluation

LLMs as a Jury: Cross-Model Consensus Can Outperform Process Reward Models for LLM Reasoning

Judge, Retrieve, or Abstain: Uncertainty-Guarded LLM Judging with Provable Risk Guarantees

When the Judge Should Not Decide: Evidence-Locked, Non-Compensatory Selection Bounds LLM-Judge Failure in Reasoning Pipelines