arXiv AI

More Agents, More Samples, or More Thinking? Untangling Coordination, Ensembling, and Test-Time Compute in Multi-Agent LLMs

The paper evaluates multi‑agent language models (LLMs) by comparing them to single‑pass baselines across nine empirical studies. It finds that multi‑agent systems generally do not outperform single‑pass baselines, with negative or near‑zero accuracy differences when matched on sample count or resource budget. The authors conclude that communication alone offers no advantage unless agents have partitioned evidence, external verification, or diverse errors, and they provide a reviewer checklist and accounting tools.

arXiv AI
Oct 21

The More the Better? A Comparison of Single- and Multi-Agent LLM Setups on a Financial Review Task

The study compares single-agent and multi-agent large language model (LLM) setups for reviewing credit assessment memos against specified requirements. In a controlled experiment with 300 synthetic memos, a single agent examined six requirements in one call, while a team of three agents each checked two requirements. Results show the team identified 0.08 to 0.10 more defects on certain models but also produced more false alarms, used 2.2 to 2.4 times more tokens, and had mixed effects on balanced accuracy depending on instruction and model.

By Pei Liu
arXiv AI
Oct 12

Do More Agents Help? Evaluating Accuracy and Cost in LLM Multi-Agent Workflows

The paper introduces BenchAgent, a framework for evaluating multi‑agent large language model (LLM) workflows against a single‑agent baseline, BenchAgent Core. Using GPT‑4.1 across ten reasoning, coding, and tool‑use benchmarks, the study compares six multi‑agent configurations, finding that EvoAgent achieves the highest mean pooled accuracy (79.70%) while other agents lag by 1.71–8.76 percentage points. The analysis also highlights trade‑offs in token consumption and accuracy, offering guidance for selecting and optimizing LLM multi‑agent workflows.

By Yuhang Fu, Ruishan Fang, Jiaqi Shao, Huiyu Zheng, Zhengtao Zhu, Bing Luo, Zhenyu Wen, Tao Lin
arXiv AI
Sep 30

Beyond Symmetric Agents: Cognitive Diversity and Multi-Agent Debate in Small Language Models

The study evaluates multi‑agent debate (MAD) in small language models, testing whether cognitive diversity—via personas, sampling temperature, or model identity—drives performance gains. Across 23 models, five tasks, and over 5,500 runs, MAD consistently outperforms single‑model inference but, when matched for generation budget, it ties or falls behind self‑consistency sampling, with persona prompting actually reducing accuracy. The authors find that MAD’s benefits largely stem from the first answer exchange and that many reported gains are due to ensemble‑sampling effects rather than true diversity, highlighting the need for budget‑matched, contamination‑checked baselines. whyItMatters":"The findings clarify that MAD’s perceived advantages may be overestimated and that future debate mechanisms must be evaluated against rigorous, budget‑matched baselines to ensure genuine performance improvements."

By Leonardo Ferreira, Gardenia Liu, Kaden Zheng
arXiv AI
Oct 2

A Multi-Agent LLM Framework for Personalized Health Checkup Interpretation and Guidance

The paper introduces a multi‑agent large language model framework that interprets personalized health checkup results by assigning distinct tasks to specialized agents and combining their outputs. Experiments on 120 Korean compound queries show that the multi‑agent approach improves overall quality scores and user preference compared to a single‑agent baseline, though it incurs higher latency and cost. The benefits are most pronounced for queries requiring personal‑record lookup.

By HyungJun Kim, Taehan Lee, Soojin Cheon
arXiv AI
Oct 26

Same Context, Different Agents: A Behavioral Study of Semantic Layers, Multi-Agent Orchestration and Memory Compaction in Data Agents

arXiv:2610.20959v2 Announce Type: replace-cross Abstract: Answering questions over heterogeneous data means combining evidence from files, databases, documents and the web. Data agents automate this,...

By Zhivar Sourati, Marianne Menglin Liu, Zheng Wang, Rongguang Wang, Sai Ashish Somayajula, Tao Sheng, Kyu J. Han, Sujith Ravi, Dan Roth
arXiv AI
3d ago

Shared Item-Level Error Limits the Aggregation of Small Language Models in Fermi Estimation

The study examines how shared item‑level error limits the benefit of aggregating small language models in Fermi estimation tasks. By decomposing log10 errors into item‑specific and residual components across 2.51 million answers from 15 open‑weight models, it finds that item‑level error accounts for about 91% of variance, reducing the expected error gain from 74–79% to only 1.7–5.4% when using teams of thirty. Additional techniques such as debate, model mixing, and temperature adjustments offer modest improvements, but no method surpasses the most accurate single model.

By Carolina Fortuna, Blaz Bertalanic
arXiv Computation and Language
Sep 15

When Agents Slow Down: Understanding LLM Agents' Test-Time Strategies via Elo-per-token Analysis

arXiv:2609.15309v1 Announce Type: new Abstract: Large language model (LLM) agents allocate test-time compute adaptively as they revise solutions, use tools, explore alternatives, and decide when to s...

By Kaiyuan Liu, Qiuyang Mang, Bo Peng, Wenhao Chai, Hanchen Li, Shreyas Pimpalgaonkar, Luke Zettlemoyer, Alex Dimakis, Alvin Cheung