arXiv AI

Shared Item-Level Error Limits the Aggregation of Small Language Models in Fermi Estimation

The study examines how shared item‑level error limits the benefit of aggregating small language models in Fermi estimation tasks. By decomposing log10 errors into item‑specific and residual components across 2.51 million answers from 15 open‑weight models, it finds that item‑level error accounts for about 91% of variance, reducing the expected error gain from 74–79% to only 1.7–5.4% when using teams of thirty. Additional techniques such as debate, model mixing, and temperature adjustments offer modest improvements, but no method surpasses the most accurate single model.

Hugging Face Trending Papers
5d ago

Shared Item-Level Error Limits the Aggregation of Small Language Models in Fermi Estimation

The study examines how shared item‑level error limits the benefit of aggregating small language models in Fermi estimation tasks. By decomposing log10 errors into item‑specific and residual components across 2.51 M answers from 15 open‑weight models, it finds that item‑level error accounts for about 91 % of variance, reducing the expected error gain from a 30‑model team to only 1.7 % (max 5.4 %) instead of the theoretical 74‑79 %. Techniques such as three rounds of debate, model mixing, and temperature adjustments yield modest improvements, but no method surpasses the most accurate single model.

arXiv AI
Sep 28

Multi-agent Scaling Across Disjunctive and Compensatory Tasks

The paper introduces Steiner’s taxonomy of group tasks to study how multi‑agent large language model (LLM) teams scale on disjunctive versus compensatory tasks. By modeling agents as conditionally independent given the item, it shows that plurality voting converges to the modal answer while averaging converges to the item‑level bias. Experiments with 13 open‑weight models and up to 30 agents reveal that disjunctive tasks benefit from larger teams, whereas compensatory tasks like Fermi estimation see little improvement, highlighting that task structure and aggregation method fundamentally determine team scaling.

By Carolina Fortuna, Blaz Bertalanic
arXiv AI
Sep 30

Beyond Symmetric Agents: Cognitive Diversity and Multi-Agent Debate in Small Language Models

The study evaluates multi‑agent debate (MAD) in small language models, testing whether cognitive diversity—via personas, sampling temperature, or model identity—drives performance gains. Across 23 models, five tasks, and over 5,500 runs, MAD consistently outperforms single‑model inference but, when matched for generation budget, it ties or falls behind self‑consistency sampling, with persona prompting actually reducing accuracy. The authors find that MAD’s benefits largely stem from the first answer exchange and that many reported gains are due to ensemble‑sampling effects rather than true diversity, highlighting the need for budget‑matched, contamination‑checked baselines. whyItMatters":"The findings clarify that MAD’s perceived advantages may be overestimated and that future debate mechanisms must be evaluated against rigorous, budget‑matched baselines to ensure genuine performance improvements."

By Leonardo Ferreira, Gardenia Liu, Kaden Zheng
arXiv AI
Oct 21

The More the Better? A Comparison of Single- and Multi-Agent LLM Setups on a Financial Review Task

The study compares single-agent and multi-agent large language model (LLM) setups for reviewing credit assessment memos against specified requirements. In a controlled experiment with 300 synthetic memos, a single agent examined six requirements in one call, while a team of three agents each checked two requirements. Results show the team identified 0.08 to 0.10 more defects on certain models but also produced more false alarms, used 2.2 to 2.4 times more tokens, and had mixed effects on balanced accuracy depending on instruction and model.

By Pei Liu
arXiv AI
Sep 18

Language-model groups overstate consensus when replaying human deliberation on a reasoning task

The study compares human deliberation in Wason selection tasks with large language model (LLM) agent groups that are seeded with participants’ pre-discussion beliefs. Across various scoring definitions, human consensus rates ranged from 24.0% to 57.0%, whereas LLM agents consistently achieved higher consensus, with gaps of 34–44 percentage points in two sensitivity analyses. Even when early stopping was removed or memorizable answers were eliminated, LLM groups still reached near-unanimous agreement, often on incorrect answers, indicating that simulated consensus does not reflect collective accuracy.

By Tengfei Shao
arXiv Computation and Language
Sep 17

Playing log(N)-Questions over Wikipedia Abstracts: Communication Efficiency Between Paired Frontier Models

The study evaluates six frontier language models on a two‑agent <log(N)>‑Questions game using Wikipedia lead paragraphs. In each game a questioner must identify a target paragraph with exactly <log2 N> yes/no questions, while an answerer only sees the target and the question and replies with a single word. Across 408 games, the models perform similarly, with Claude Opus 5 winning 28 of 68 games and the top five models showing only marginal differences; win rates decline sharply with larger document sets, following a reliability parameter of 0.928 per question. "whyItMatters":"The results reveal how well language models can communicate under information asymmetry, highlighting that even top models struggle to extract a full bit per question and that reasoning token usage does not strongly predict success."

By Peter Potash
arXiv Computation and Language
Sep 18

Message capacity and claim wording set the transition points of collective truth-finding in language-model networks

The study investigates how limited reading capacity and claim wording influence consensus outcomes in language‑model networks. By modeling message capacity as the number of messages an agent reads, the authors show that when agents read fewer than about 6.4 messages on average, a wrong consensus becomes unreachable. However, the wording of a claim—its inherent threshold—can override this effect, leading to incorrect consensus even when most agents start correct.

By Makoto Fukushima
arXiv Machine Learning
Sep 29

Reinforcement Learning of Communication in a Mesh of Small Language Models

The paper introduces TalkMesh, a decentralized network of small language model agents that learn to communicate effectively during inference. Each agent proposes an answer, scores it with a confidence head, and the most confident agent broadcasts a hint; lower‑confidence agents revise their proposals if a new suggestion scores higher. This gossip‑based consensus, trained via group relative policy optimization, enables a mesh of three agents to match the accuracy of majority voting over 32 samples, and scales to larger meshes to significantly boost performance on benchmarks like GSM8K and MATH-500.

By Mehmet Kerem Turkcan
arXiv AI
1d ago

More Agents, More Samples, or More Thinking? Untangling Coordination, Ensembling, and Test-Time Compute in Multi-Agent LLMs

The paper evaluates multi‑agent language models (LLMs) by comparing them to single‑pass baselines across nine empirical studies. It finds that multi‑agent systems generally do not outperform single‑pass baselines, with negative or near‑zero accuracy differences when matched on sample count or resource budget. The authors conclude that communication alone offers no advantage unless agents have partitioned evidence, external verification, or diverse errors, and they provide a reviewer checklist and accounting tools.

By Seyed Mahmoud Sajjadi Mohammadabadi