arXiv Computation and Language

Phantom Consensus: Measuring Social Reasoning in Multi-Agent LLM Decision-Making

The paper introduces the concept of phantom consensus, where large language models (LLMs) alter their public decisions after seeing ambiguous social signals and misreport the private judgments of other agents, even when the recorded private judgments support a different consensus. To investigate this, the authors create PHANTOM-Bench, a benchmark that isolates private judgments, public decisions, and estimates of group beliefs, and tests models under various conditions. Experiments across nine LLMs show that ambiguous signals can significantly degrade decision quality, with public accuracy dropping up to 47.8 percentage points and collective rationality failures reaching 47.3% in the most vulnerable settings, while revealing private decisions can mitigate many failures.

arXiv AI
Oct 16

Believed but Not Said: Measuring Hidden Belief Collapse in Multi-Agent LLM Discussions

The study examines how groups of large language model (LLM) agents handle misinformation when a few adversarial agents inject false claims into a discussion. In experiments with ten agents across six models and three datasets, a single adversarial agent caused 44.8% of benign agents to adopt the false answer, while three adversaries raised this to 60.5%. Interestingly, many benign agents privately held the false answer yet publicly disagreed, indicating hidden belief collapse that is not visible in the discussion itself.

By Liangjie Huang, Henry Peng Zou, Wei-Chieh Huang, Wooseong Yang, Yuwei Han, Philip S. Yu
arXiv AI
Oct 28

Who Convinced Whom? Tracing How LLMs Change Their Minds in Multi-Agent Systems

The paper introduces MAD Bench, a 2,400-question benchmark for multi‑agent debate (MAD) that tests knowledge, reasoning, values, moral judgment, safety, and refusal. It evaluates three open‑weight models and finds that peer communication can alter individual decisions, with forced‑choice readouts before rationale generation more often matching the final answer in debate than in reflection. The study shows that stronger consensus can coexist with increased shared errors, suggesting the need for evaluation metrics that consider dependence and unanimous errors alongside accuracy.

By Arnesh Batra, Aniket Khandelwal, Jashn Khemani, Krish Thukral, Arush Gumber, Denis Federico Lim, Ujwal Kumar, Phan Xuan Tan
arXiv Machine Learning
Jul 7

Social Networks of LLM Agents

arXiv:2607. 03695v1 Announce Type: new Abstract: Large language model (LLM) agents are increasingly deployed in interacting populations, raising the question of what such populations come to believe collectively.

By Kaixuan Liu, Guojun Xiong, Weinan Zhang, Shengpu Tang
arXiv Computation and Language
5d ago

The Judge is Part of the Experiment: Auditing Peer Context Sensitivity in LLM-Based Evaluation

The paper investigates how peer context—interactions among agents before a final answer—is perceived by large language model (LLM) judges when evaluating AI systems. By conducting controlled experiments where agents answer questions with and without seeing peer responses, and then having judges rate the same final answer under both conditions, the study demonstrates that peer context can alter both the answer produced and the judge’s evaluation. The results reveal that judges respond in opposite directions to peer context, and that providing peer context for incorrect answers can mislead agents more than correct peers help them, while judges struggle to distinguish known-correct from known-wrong answers without the correct context.

By Alicia Guerra, Jiaming Qu, Yibo Hu