arXiv AI
Jul 3
GroundEval: A Deterministic Replacement for LLM-as-Judge in Stateful Agent Evaluation
arXiv:2606. 22737v2 Announce Type: replace Abstract: Before letting an agent operate over real context, can you prove it used the right evidence?
Grounded but Not Immune: Confirmation Bias in Agent-as-a-judge
The paper investigates confirmation bias in Agent-as-a-Judge systems, which evaluate open-ended agent tasks by gathering evidence from the agent’s trajectory. It demonstrates that a single false pre‑screening note can overturn up to 40% of correct verdicts across multiple judges and datasets, even when the judge collects decisive evidence. The authors classify the resulting errors into over‑trust, hallucination, and under‑investigation, noting that over‑trust dominates in larger Claude judges, and show that lightweight mitigations reduce but do not eliminate the bias.
Related stories
arXiv Computer Vision
Oct 2
A Low Grounding Score Is Not an Ungrounded Judge: Identifying the Perceptibility Confound in Multimodal Oversight
arXiv:2610.00111v1 Announce Type: new Abstract: Model judges now supervise multimodal systems at scale, filtering training data, selecting outputs, and supplying the reward that shapes multimodal rea...
Hugging Face Trending Papers
Jun 22
GroundEval: A Deterministic Replacement for LLM-as-Judge in Stateful Agent Evaluation
Before letting an agent operate over real context, can you prove it used the right evidence? GroundEval turns that question into a deterministic test of what the agent searched, fetched, cited, and was permitted to access.
arXiv AI
Sep 28
When Is a Multi-Agent Code Judge Actually Grounded? Two Label-Free Measurements, and a Judge That Declines to Guess
The paper investigates when a language‑model judge can truly ground its verdicts in code correctness. It shows that current multi‑agent verification methods rely on evidence that is both independent of the answer and distinct between candidates—conditions that fail in code judging. By analyzing two label‑free measurements from the judge’s logs, the authors demonstrate that gating on one measurement allows the system to decline uncertain comparisons, improving accuracy from 20.7% to 36.9% while still answering half of all cases.
arXiv Machine Learning
Oct 23
What Lies Beyond Perception: A Partition Theory of the Limits of AI Evaluation and Oversight
The paper introduces a partition theory of AI evaluation, arguing that an evaluator’s ability to certify a model’s capabilities is limited by its perception. It shows that if a model’s input partition refines the evaluator’s, it can pass all tests the evaluator can grade, but only a stricter evaluator can raise the discrimination floor. The authors define an echo test that is complete for the evaluator’s tests, proving that passing it only guarantees a minimum discrimination level and cannot detect exact imitation of weaker models.
arXiv AI
Aug 26
More Rejective, Not More Discriminative: The Unit of Verification in Pre-Execution LLM Oversight
The paper introduces the twin‑prefix framework to evaluate how the size of the verification unit—i.e., how many actions a pre‑execution LLM monitor reviews in one call—affects its performance. By pairing each gold plan with a twin that differs by a single write and injecting a controlled error, the authors isolate the impact of review length on catch rates and false rejections. Their findings show that longer review windows increase rejection rates but do not improve discrimination, with the highest informedness occurring at one or two actions across all judges and domains.
arXiv Computation and Language
Oct 22
Conflicted Records: Typed Adversarial Contracts for Auditable LLM Output Quality Enforcement
The paper introduces BiasGuard‑D, a system that uses two adversarial language‑model agents from different families to exchange typed claim documents, with a third model quorum voting to adjudicate. A combined referential‑plus‑semantic invariant validates the verdict, and the iterative deliberation layer shifts remediation from stereotypes to hallucinations, improving right‑answer recovery by 49 percentage points on HaluEval QA. All claims, votes, and instructions are stored as typed records, providing an auditable provenance trail for operators under emerging AI regulations.
Hugging Face Trending Papers
Oct 21
Conflicted Records: Typed Adversarial Contracts for Auditable LLM Output Quality Enforcement
LLM-generated records increasingly populate production information systems where they are treated as data assets subject to the same quality expectations as records from operational databases. A singl...
arXiv AI
Jul 29
When Do Agent Loops Mistake Stagnation for Progress? Self-Evaluation Bias and Externally Grounded Verification in Long-Running Autonomous LLM Agent Loops
arXiv:2607. 25152v1 Announce Type: new Abstract: Long-running autonomous agents plan, act, and judge their own completion without human intervention.
arXiv AI
Aug 24
JuryProbe: An Empirical Consensus-Risk Diagnostic for Routing Reference-Free Factuality Judge Panels to Grounded Verification
JuryProbe is an empirical diagnostic tool designed to assess consensus risk in panels of reference‑free large language model judges used for factuality verification. It estimates risk by measuring false‑negative correlations and false‑consensus lift from a labeled calibration probe, and routes high‑risk majority decisions to judges with trusted references. The approach was validated on FEVER corruptions, showing that flagged decisions can be grounded without additional reference acquisition in most cases, while reducing false accepts by about 0.4% and avoiding 28% of reference acquisitions.
arXiv AI
Jun 16
Who Drifted: the System or the Judge? Anytime-Valid Attribution in LLM Evaluation Pipelines
arXiv:2606. 15474v1 Announce Type: new Abstract: Continuous evaluation of LLM products relies on a strong LLM judge treated as ground truth: a cheap monitor scores every interaction and a team is paged when the score drifts down.
arXiv AI
Aug 14
Jagged Judges: Epistemic Stability Under Silence, Pressure, and Persistence
arXiv:2608. 12645v1 Announce Type: new Abstract: LLM judges have become central infrastructure for model evaluations, online grading, and reward modeling.