The Judge is Part of the Experiment: Auditing Peer Context Sensitivity in LLM-Based Evaluation
Read the original on arXiv Computation and Language →The paper investigates how peer context—interactions among agents before a final answer—is perceived by large language model (LLM) judges when evaluating AI systems. By conducting controlled experiments where agents answer questions with and without seeing peer responses, and then having judges rate the same final answer under both conditions, the study demonstrates that peer context can alter both the answer produced and the judge’s evaluation. The results reveal that judges respond in opposite directions to peer context, and that providing peer context for incorrect answers can mislead agents more than correct peers help them, while judges struggle to distinguish known-correct from known-wrong answers without the correct context.
Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computation and Language.