The paper investigates how peer context—interactions among agents before a final answer—is perceived by large language model (LLM) judges when evaluating AI systems. By conducting controlled experiments where agents answer questions with and without seeing peer responses, and then having judges rate the same final answer under both conditions, the study demonstrates that peer context can alter both the answer produced and the judge’s evaluation. The results reveal that judges respond in opposite directions to peer context, and that providing peer context for incorrect answers can mislead agents more than correct peers help them, while judges struggle to distinguish known-correct from known-wrong answers without the correct context.
arXiv:2608.25937v2 Announce Type: replace Abstract: Multi-agent systems (MAS) sometimes already have the potential to answer correctly, but still report a wrong answer. Explaining this outcome is dif...
arXiv:2602.02589v2 Announce Type: replace Abstract: Large language model (LLM) evaluation commonly relies on fixed human-authored benchmarks or centralized human/model judges. Fixed benchmarks can ag...
arXiv:2606. 22737v2 Announce Type: replace Abstract: Before letting an agent operate over real context, can you prove it used the right evidence?
arXiv:2606. 01637v1 Announce Type: cross Abstract: Large language models are increasingly used in multi-agent systems, where they see and respond to other agents' answers.
The paper introduces MAD Bench, a 2,400-question benchmark for multi‑agent debate (MAD) that tests knowledge, reasoning, values, moral judgment, safety, and refusal. It evaluates three open‑weight models and finds that peer communication can alter individual decisions, with forced‑choice readouts before rationale generation more often matching the final answer in debate than in reflection. The study shows that stronger consensus can coexist with increased shared errors, suggesting the need for evaluation metrics that consider dependence and unanimous errors alongside accuracy.
Before letting an agent operate over real context, can you prove it used the right evidence? GroundEval turns that question into a deterministic test of what the agent searched, fetched, cited, and was permitted to access.
arXiv:2608. 07762v1 Announce Type: new Abstract: LLM benchmarks can build an organization's reputation and attract customers, but only when results are transparent and verifiable.
JudgeSense is a benchmark comprising 880 items from human‑labelled corpora, each presented under two differently worded instructions that ask the same question. The study evaluates 25 judges from six providers across four tasks, measuring how rewording affects agreement with the judge’s own verdicts. Results show that rewording reduces agreement on all tasks, with significant effects on two, and that stability varies across tasks and is not predicted by parameter count.
The paper investigates confirmation bias in Agent-as-a-Judge systems, which evaluate open-ended agent tasks by gathering evidence from the agent’s trajectory. It demonstrates that a single false pre‑screening note can overturn up to 40% of correct verdicts across multiple judges and datasets, even when the judge collects decisive evidence. The authors classify the resulting errors into over‑trust, hallucination, and under‑investigation, noting that over‑trust dominates in larger Claude judges, and show that lightweight mitigations reduce but do not eliminate the bias.
The paper investigates when a language‑model judge can truly ground its verdicts in code correctness. It shows that current multi‑agent verification methods rely on evidence that is both independent of the answer and distinct between candidates—conditions that fail in code judging. By analyzing two label‑free measurements from the judge’s logs, the authors demonstrate that gating on one measurement allows the system to decline uncertain comparisons, improving accuracy from 20.7% to 36.9% while still answering half of all cases.
arXiv:2610.15938v1 Announce Type: new Abstract: Evaluation of text generated by large language models (LLMs) is increasingly delegated to LLMs acting as judges, yet what such a judge must be shown re...
The paper introduces FoG, a framework that trains knowledge graph (KG) agents to retrieve the evidence behind their answers. FoG combines evidence‑grounded synthesis, a KG interaction agent, and an evidence‑coverage process reward (ECPR) to ensure that every answer is supported by retrieved triples. Experiments on three KGQA benchmarks show that 3B–4B open‑source agents trained with FoG outperform larger baselines in both answer accuracy and evidence coverage, even with zero‑shot cross‑KG transfer.