arXiv AI By Carolina Fortuna, Blaz Bertalanic

Shared Item-Level Error Limits the Aggregation of Small Language Models in Fermi Estimation

Read the original on arXiv AI →

The study examines how shared item‑level error limits the benefit of aggregating small language models in Fermi estimation tasks. By decomposing log10 errors into item‑specific and residual components across 2.51 million answers from 15 open‑weight models, it finds that item‑level error accounts for about 91% of variance, reducing the expected error gain from 74–79% to only 1.7–5.4% when using teams of thirty. Additional techniques such as debate, model mixing, and temperature adjustments offer modest improvements, but no method surpasses the most accurate single model.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

Hugging Face Trending Papers
5d ago

Shared Item-Level Error Limits the Aggregation of Small Language Models in Fermi Estimation

The study examines how shared item‑level error limits the benefit of aggregating small language models in Fermi estimation tasks. By decomposing log10 errors into item‑specific and residual components across 2.51 M answers from 15 open‑weight models, it finds that item‑level error accounts for about 91 % of variance, reducing the expected error gain from a 30‑model team to only 1.7 % (max 5.4 %) instead of the theoretical 74‑79 %. Techniques such as three rounds of debate, model mixing, and temperature adjustments yield modest improvements, but no method surpasses the most accurate single model.

arXiv AI
Sep 28

Multi-agent Scaling Across Disjunctive and Compensatory Tasks

The paper introduces Steiner’s taxonomy of group tasks to study how multi‑agent large language model (LLM) teams scale on disjunctive versus compensatory tasks. By modeling agents as conditionally independent given the item, it shows that plurality voting converges to the modal answer while averaging converges to the item‑level bias. Experiments with 13 open‑weight models and up to 30 agents reveal that disjunctive tasks benefit from larger teams, whereas compensatory tasks like Fermi estimation see little improvement, highlighting that task structure and aggregation method fundamentally determine team scaling.

By Carolina Fortuna, Blaz Bertalanic
arXiv AI
Sep 30

Beyond Symmetric Agents: Cognitive Diversity and Multi-Agent Debate in Small Language Models

The study evaluates multi‑agent debate (MAD) in small language models, testing whether cognitive diversity—via personas, sampling temperature, or model identity—drives performance gains. Across 23 models, five tasks, and over 5,500 runs, MAD consistently outperforms single‑model inference but, when matched for generation budget, it ties or falls behind self‑consistency sampling, with persona prompting actually reducing accuracy. The authors find that MAD’s benefits largely stem from the first answer exchange and that many reported gains are due to ensemble‑sampling effects rather than true diversity, highlighting the need for budget‑matched, contamination‑checked baselines. whyItMatters":"The findings clarify that MAD’s perceived advantages may be overestimated and that future debate mechanisms must be evaluated against rigorous, budget‑matched baselines to ensure genuine performance improvements."

By Leonardo Ferreira, Gardenia Liu, Kaden Zheng
arXiv AI
Oct 21

The More the Better? A Comparison of Single- and Multi-Agent LLM Setups on a Financial Review Task

The study compares single-agent and multi-agent large language model (LLM) setups for reviewing credit assessment memos against specified requirements. In a controlled experiment with 300 synthetic memos, a single agent examined six requirements in one call, while a team of three agents each checked two requirements. Results show the team identified 0.08 to 0.10 more defects on certain models but also produced more false alarms, used 2.2 to 2.4 times more tokens, and had mixed effects on balanced accuracy depending on instruction and model.

By Pei Liu
arXiv AI
Sep 18

Language-model groups overstate consensus when replaying human deliberation on a reasoning task

The study compares human deliberation in Wason selection tasks with large language model (LLM) agent groups that are seeded with participants’ pre-discussion beliefs. Across various scoring definitions, human consensus rates ranged from 24.0% to 57.0%, whereas LLM agents consistently achieved higher consensus, with gaps of 34–44 percentage points in two sensitivity analyses. Even when early stopping was removed or memorizable answers were eliminated, LLM groups still reached near-unanimous agreement, often on incorrect answers, indicating that simulated consensus does not reflect collective accuracy.

By Tengfei Shao