arXiv AI
Agents-as-Peers: Bias-Controlled LLM Evaluation as Multi-Agent Peer Assessment
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv Computation and Language
Sep 1
Beyond Consensus: Downward Bias and Role Asymmetry in Multi-Agent LLM Judges for Subjective Evaluation
arXiv:2608.30373v1 Announce Type: new Abstract: Multi-Agent Debate (MAD) has been widely adopted to improve LLM-based evaluation by prompting multiple agents to negotiate and reach a consensus. Howev...
arXiv AI
Aug 11
In-Situ Behavioral Evaluation for LLM Fairness, Not Standardized-Test Scores
arXiv:2605. 12530v2 Announce Type: replace-cross Abstract: LLM fairness should be evaluated through in-situ behavioral pattern rather than standardized-test Q&A benchmarks.
Hugging Face Trending Papers
Aug 31
Beyond Consensus: Downward Bias and Role Asymmetry in Multi-Agent LLM Judges for Subjective Evaluation
Multi-Agent Debate (MAD) has been widely adopted to improve LLM-based evaluation by prompting multiple agents to negotiate and reach a consensus. However, for subjective rubric-based scoring, inter-ag...
arXiv AI
Aug 11
Who Verifies the Benchmark? Decentralizing Trust in Large Language Model Evaluation
arXiv:2608. 07762v1 Announce Type: new Abstract: LLM benchmarks can build an organization's reputation and attract customers, but only when results are transparent and verifiable.
arXiv Machine Learning
Sep 14
GAUGE: When Not to Trust LLM-as-a-Judge in User-Simulated Evaluation of Task-Oriented Agents
GAUGE is a new offline protocol that evaluates whether the common practice of using an LLM-as-a-judge to rank task‑oriented agents actually aligns with a verifiable reward. Across 25 agents from six providers on two benchmarks, GAUGE finds that user satisfaction scores are largely uncorrelated with task success, and that the judge’s ranking loses precision when agents are closely matched in performance. The study highlights a gap between ranking validity and construct validity in current evaluation practices.
arXiv AI
Aug 24
Peer-Voted LLM-Agent Stress Tests Find Feed-Induced Lexical Convergence but No Reliable Matched-Exposure Advantage for Distributed Sources
The study introduces PV‑SST, a peer‑voted social‑platform testbed, and conducts a preregistered matched‑exposure experiment across four topics, four seeds, four model families, and three larger variants, totaling 448 trials. Results show that feeding agents a ranked list of prior peer posts increases lexical similarity in both the core panel and larger variants, but does not produce a reliable advantage in opinion alignment or survival rates. The only robust finding is lexical convergence driven by the peer‑ranked feed, with no consistent coordination benefit across models or topics.