arXiv Computation and Language

From Diagnosis to Redesign: Using Quantitative Ethnography to Improve Multi-Agent LLM Reasoning

arXiv AI
Jun 2

Demystifying Multi-Agent Debate: The Role of Confidence and Diversity

arXiv:2601. 19921v2 Announce Type: replace-cross Abstract: Multi-agent debate (MAD) is widely used to improve large language model (LLM) performance through test-time scaling, yet recent work shows that vanilla MAD often underperforms simple majority vote despite higher computational cost.

By Xiaochen Zhu, Caiqi Zhang, Yizhou Chi, Tom Stafford, Nigel Collier, Andreas Vlachos
arXiv AI
Aug 20

A Theory of Post-hoc Debate Judgement

The paper proposes a theory for judging post-hoc debates in AI, focusing on properties like reproducibility, robustness, groundedness, and explainability. It evaluates two debate‑judgement methods—LLM judges and formal computational argumentation semantics—finding similar accuracy but noting that argumentation semantics offers stronger formal guarantees. The study suggests that argumentation semantics is a preferable framework for principled debate judges in AI systems.

By Xiang Yin, Adam Dejl, Antonio Rago, Lihu Chen, Francesca Toni
arXiv AI
Sep 10

How AI Models Manage Epistemic Authority: A Taxonomy and Comparative Analysis of Responses to User Disagreement

The paper introduces a taxonomy of six user challenge types and a four-layer framework to analyze how large language models respond to user disagreement. Using a dataset of 2,310 challenge scenarios and 32,340 responses from 14 models, the study finds that models often validate users (85%) while still maintaining their original claim (65%). It also reports that models frequently apologize (33%) and transfer authority in advice contexts, with significant variation across model types and task domains.

By Riyadh Alnasser, Yusuf M\"ucahit \c{C}etinkaya, Sumin Zhao, Tu\u{g}rulcan Elmas