arXiv AI
Jul 10
When LLMs Agree, Are They Right? Auditing Self-Consistency and Cross-Model Agreement as Confidence Signals
arXiv:2607. 08065v1 Announce Type: new Abstract: LLM-as-judge (Zheng et al.
A latent dimension of Condorcet's jury theorem for multiple AI advisers
Related stories
arXiv AI
Sep 10
A Layered Analysis of Disagreement And Answer Quality in Multi-Agent LLM Debate
arXiv:2609.08016v1 Announce Type: new Abstract: Multi-agent debate, in which several LLMs exchange arguments before answering, is widely assumed to improve answer quality by surfacing genuine disagre...
arXiv AI
Sep 23
Agreement Overstates Evidence: Error Dependence in LLM Judge Consensus
arXiv:2609.22512v1 Announce Type: new Abstract: Consensus among LLM judges is often taken as strong evidence that a decision is correct. This assumes that judges make their errors independently. In p...
arXiv Computation and Language
Oct 19
Same Person, Different Advice: Auditing LLM Financial Advice with Repeated Risk-Tolerance Responses
The study examines how large language models (LLMs) provide financial advice based on a single risk‑tolerance question, finding that both human respondents and LLMs exhibit significant variability when the same question is repeated. In a national US panel, 41% of households changed their risk answer after two years, and in a lab retest a third changed by two or more points on a 0‑10 scale after six months. When LLMs were given 800 real household profiles with only the risk answer altered, the recommended stock allocation shifted by 10–16 percentage points on average, and by 24–35 points when the answer changed, compared to minimal swings when the same prompt was repeated. The study also shows that adding clarifying questions or noting answer noise can reduce these swings, especially for younger clients and those with extreme risk attitudes. whyItMatters":"The findings highlight that LLMs can produce markedly different financial advice from a single, potentially unreliable risk‑tolerance response, underscoring the need for repeat‑testing and transparency in AI‑driven financial recommendations."
arXiv AI
Sep 7
Measuring AI Accountability Through Argumentation Analysis: Can Model Reasoning Withstand Scrutiny?
The paper proposes a new method for evaluating AI accountability by analyzing the structural quality of a model’s defense for its decisions, using a four‑phase dialectical protocol based on Walton’s argumentation schemes and Govier’s criteria. Applied to nine large language models and 200 ambiguous moral-choice items, the study finds that models generally defend their reasoning well above the rubric minimum, though failures cluster on grounds and sufficiency and correlate with epistemic hedging. The protocol also reveals that models often present different argument schemes in justification than in reasoning, detects indefensible defenses, and highlights challenges in assessing retraction in AI alignment.
arXiv Computation and Language
Oct 1
Multi-agent discussion gains less when dissent is withheld
arXiv:2609.38324v1 Announce Type: cross Abstract: Multi-agent systems of LLMs add discussion to majority voting and are therefore expected to be more capable. However, empirical reports conflict on w...
arXiv Machine Learning
Oct 19
Think More or Read More? A Token-Budget Theory of Reasoning with Shared Context
arXiv:2610.23246v1 Announce Type: cross Abstract: Reasoning systems increasingly improve answers by incorporating additional context into the model's input, from multi-agent debate, where agents read...
arXiv AI
Oct 21
Humans Correct Their Judgments Through Debate, but Weaker AI Models Do Not
The paper examines whether AI‑driven debate can help humans and weaker language models (LLMs) correct their judgments. Experiments show that debate improves human accuracy over prior beliefs, especially with weighted voting, but offers little benefit—and sometimes harms—accuracy for weaker LLM supervisors. The study highlights debate as a promising tool for scalable human oversight while questioning its effectiveness for fully automated AI control.
arXiv AI
Jun 2
Demystifying Multi-Agent Debate: The Role of Confidence and Diversity
arXiv:2601. 19921v2 Announce Type: replace-cross Abstract: Multi-agent debate (MAD) is widely used to improve large language model (LLM) performance through test-time scaling, yet recent work shows that vanilla MAD often underperforms simple majority vote despite higher computational cost.
arXiv AI
Oct 2
Counting Moves, Weighing Voices: Bayesian Dialectical Argumentation for Calibrated Multi-LLM Councils under Persistent Adversaries
The paper introduces Bayesian Dialectical Argumentation (BDA), a method for aggregating answers from multiple large language models (LLMs) in a council setting. BDA treats each LLM’s typed moves—proposals, challenges, and concessions—as evidence in a classical annotator model, estimating per-agent reliability even when some agents are persistently unreliable. By weighting evidence according to these inferred reliabilities, BDA produces calibrated posterior probabilities for candidate answers and can invert unreliable agents instead of merely outvoting them, achieving superior calibration and robustness on both binary and multi-class benchmarks without extra LLM calls.
arXiv AI
Jun 15
The Coin Flip Judge? Reliability and Bias in LLM-as-a-Judge Evaluation
arXiv:2606. 13685v1 Announce Type: cross Abstract: LLM-as-a-Judge is now widely used to rank model outputs, train reward models, and populate public leaderboards, but its run-to-run reliability remains under-characterized.
arXiv AI
Sep 1
Candidate supply and answer selection shape the value of LLM judging in multi-agent systems
arXiv:2608.25937v2 Announce Type: replace Abstract: Multi-agent systems (MAS) sometimes already have the potential to answer correctly, but still report a wrong answer. Explaining this outcome is dif...