The study examines how shared item‑level error limits the benefit of aggregating small language models in Fermi estimation tasks. By decomposing log10 errors into item‑specific and residual components across 2.51 million answers from 15 open‑weight models, it finds that item‑level error accounts for about 91% of variance, reducing the expected error gain from 74–79% to only 1.7–5.4% when using teams of thirty. Additional techniques such as debate, model mixing, and temperature adjustments offer modest improvements, but no method surpasses the most accurate single model.
By Carolina Fortuna, Blaz Bertalanic
The study examines how shared item‑level error limits the benefit of aggregating small language models in Fermi estimation tasks. By decomposing log10 errors into item‑specific and residual components across 2.51 M answers from 15 open‑weight models, it finds that item‑level error accounts for about 91 % of variance, reducing the expected error gain from a 30‑model team to only 1.7 % (max 5.4 %) instead of the theoretical 74‑79 %. Techniques such as three rounds of debate, model mixing, and temperature adjustments yield modest improvements, but no method surpasses the most accurate single model.
arXiv:2609.35928v1 Announce Type: cross
Abstract: Multi-agent LLM systems increasingly mix models from several providers, yet exposing each agent's underlying model identity to its peers significantl...
By Xavier Del Giudice, Alessio Palma, Matteo Migliarini, Fabio Galasso, Indro Spinelli
arXiv:2608.22152v1 Announce Type: new
Abstract: Multi-agent systems built from large language models are deployed widely, yet how much performance is lost when two LLMs must coordinate rather than ac...
By Weixiang Sun, Zehong Wang, Hong Huang, Colby Nelson, Yanfang Ye
The study evaluates multi‑agent debate (MAD) in small language models, testing whether cognitive diversity—via personas, sampling temperature, or model identity—drives performance gains. Across 23 models, five tasks, and over 5,500 runs, MAD consistently outperforms single‑model inference but, when matched for generation budget, it ties or falls behind self‑consistency sampling, with persona prompting actually reducing accuracy. The authors find that MAD’s benefits largely stem from the first answer exchange and that many reported gains are due to ensemble‑sampling effects rather than true diversity, highlighting the need for budget‑matched, contamination‑checked baselines.
whyItMatters":"The findings clarify that MAD’s perceived advantages may be overestimated and that future debate mechanisms must be evaluated against rigorous, budget‑matched baselines to ensure genuine performance improvements."
By Leonardo Ferreira, Gardenia Liu, Kaden Zheng
arXiv:2606. 02646v1 Announce Type: cross Abstract: Inference-time multi-agent LLM scaling lacks a shared unit: counting nominal agents conflates cost with independent evidence.
By Bla\v{z} Bertalani\v{c}, Carolina Fortuna