arXiv AI
Candidate supply and answer selection shape the value of LLM judging in multi-agent systems
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv AI
Aug 17
Wrong but Useful: Trajectory Value Beyond Answer Correctness in Multi-Agent Messages
arXiv:2608. 14375v1 Announce Type: new Abstract: Multi-agent reasoning systems often use agreement, confidence, or automated scores to decide which messages should shape a final answer.
arXiv AI
Jul 3
GroundEval: A Deterministic Replacement for LLM-as-Judge in Stateful Agent Evaluation
arXiv:2606. 22737v2 Announce Type: replace Abstract: Before letting an agent operate over real context, can you prove it used the right evidence?
arXiv AI
3d ago
Asking for What Was Never Requested: Horizontal and Vertical Proactivity in Agents
arXiv:2609.37236v1 Announce Type: new Abstract: An agent that uses tools typically responds to what the user explicitly asks, yet completing the task may require information the user never requested....
arXiv AI
Aug 28
AgentJudgeBench: A Multi-Difficulty Benchmark for Evaluating LLM Judges on Agentic Tool-Calling
AgentJudgeBench is a new benchmark that evaluates the reliability of large language model (LLM) judges on agentic tool‑calling tasks involving workflow directed acyclic graphs (DAGs). It contains 3,808 instances across six DAG topologies and three difficulty tiers, tested with five generators (3B–70B open‑weight models and GPT‑5.4) and six judges (20B to frontier scale) under both paired‑with‑and‑without‑ground‑truth conditions. The study finds that judge alignment degrades with task difficulty, ground‑truth exposure can sometimes hurt alignment, and structured evaluation rubrics provide modest improvements, revealing a structural ceiling that model capacity alone cannot surpass.
arXiv AI
Jun 16
Let LLMs Judge Each Other: Multi-Agent Peer-Reviewed Reasoning for Medical Question Answering
arXiv:2606. 15419v1 Announce Type: cross Abstract: Objective: To enhance the accuracy, interpretability, and robustness of large language models (LLMs) in medical question answering (MedQA).
arXiv AI
Jun 30
Preventing Error Propagation in Multi-Agent AI through Runtime Monitoring
arXiv:2606. 29026v1 Announce Type: new Abstract: Multi-agent AI systems can improve answer selection by allowing different language models to exchange reasoning traces, revise initial predictions, and support a final decision.