The study evaluates how to best split tasks among large‑language‑model agents for cross‑border VAT determination, comparing one broad agent to configurations ranging from one to five narrow agents. Across 4,400 runs—including token‑matched and failure‑injection scenarios—the intermediate configurations achieved the highest accuracy but did not surpass the fine‑endpoint benchmark, leaving the optimal decomposition hypothesis unconfirmed. The pilot provides a preregistered heuristic for right‑sizing decomposition, along with an oracle, dataset, and analysis pipeline.
arXiv:2607. 17531v1 Announce Type: cross Abstract: Test-time collaboration, including self-consistency, best-of-N selection, critic models, and verifier pipelines, is often credited with broadly improving LLM reasoning, yet its gains are uneven and sometimes negative.
The paper evaluates a three‑part control stack— a 5‑Pillar runtime constitution, a 4‑tier swarm (Director, majority vote, Monitor, hard gate), and Cognitive Annealing—for multi‑agent LLM negotiation against an adversarial Gatekeeper. Experiments with Gemini 2.5 Pro agents and Claude Haiku 4.5 show that the constitution and Director improve framing but not reliability, the Monitor and hard gate do not reduce unlocks, and deterministic recovery via atomic purge and a canonical strike successfully resolves deadlocks while keeping call budgets unchanged. The study demonstrates that the stack can recover from deadlocks without additional model cost, though it does not reduce overall call or token usage as initially hypothesized.
arXiv:2609.11987v1 Announce Type: cross Abstract: An agentic coding system couples a language model to a harness: the tools, prompts and control flow that turn a chat model into an autonomous softwar...
arXiv:2607. 17136v1 Announce Type: cross Abstract: Agentic computer-use RL is reported in single runs, and those numbers mislead.
The study evaluates how runtime gates for agent tool calls stack, assuming their errors multiply. Using 1,119 labeled agent actions from three corpora, the authors compare a deterministic rule layer with four LLM judges, measuring the effective number of multiplication-equivalent layers (n_mult). Results show that two judges compose to about 1.2–1.4 layers, while a rule layer plus one judge composes to roughly 1.86–2.09 layers, with variations depending on miss definitions and corpora.
arXiv:2606. 29654v1 Announce Type: new Abstract: Multi-agent deliberation among LLMs can improve reasoning, but deployment requires deciding when the current answer is reliable enough to act on and when it should be escalated to human review.
Agent evaluations tell us that a model picked the wrong tool, but rarely why. We introduce canary tools: diagnostic probe tools planted in an agent's Model Context Protocol (MCP) tool set, each engineered to probe one specific tool-selection weakness.
The paper argues that using a large language model (LLM) as the sole judge in self‑improving agent pipelines is problematic, as the judge can be biased or manipulated, leading to false confidence in system performance. The authors propose a new framework, PROCTOR, which replaces the oracle judge with a deterministic, teacher‑student loop that enforces guardrails such as sandboxing, role separation, and acceptance checks to prevent cheating and ensure reliable evaluation. Experiments across contract analysis, compliance review, and code quality demonstrate that PROCTOR mitigates eleven identified failure modes that previously allowed agents to achieve perfect scores while hiding significant capability gaps.
arXiv:2608. 11318v1 Announce Type: cross Abstract: Many sequential construction tasks exhibit exact symmetry at completion while their execution remains directed and history-dependent.
The paper evaluates a manager‑worker scaffold that uses a shared filesystem workspace to orchestrate multi‑agent large language model (LLM) coding tasks without training or tuning. Across nine models—including five open‑weight and four closed‑weight systems—the scaffold yields statistically significant accuracy gains for some models (e.g., Qwen3.8‑27B, GPT‑5.6‑Luna, GPT‑5.6‑Terra, Kimi‑K3, Minimax‑M3) while producing null or negative effects for others (e.g., Qwen3.6‑35B). The study shows that the manager can triple token usage but still achieves higher accuracy at a fraction of the cost compared to larger single‑pass models, with key mechanisms identified as context management and problem decomposition.
SheetMind is a Manager‑Action‑Reflection framework that evaluates how much spreadsheet agent performance derives from the agents themselves versus the shared action interface. In a controlled study on all 221 tasks of the SheetCopilot Benchmark, replacing the high‑level action API with primitive cell operations drops accuracy by 47.1 points, while adding a Reflection Agent improves performance by 4.5 points and a Manager by 1.4 points. The framework also shows that decomposition changes failure modes, reducing silent wrong outputs from 33% to 25%, and that GPT‑5 and GPT‑5‑mini achieve similar performance, whereas GPT‑3.5 underperforms significantly.
UnitBoost proposes a non‑generative merge operator to manage compound LLM systems, replacing the opaque higher‑level LLM that traditionally coordinates worker outputs. The operator maps worker outputs to slot‑value proposals, uses a constrained argmax to assemble the final answer, and explicitly tracks unfilled slots as residuals for subsequent rounds, thereby achieving order‑free processing and unit provenance. Across three held‑out benchmarks, UnitBoost outperforms both gold‑label‑selected candidates and input‑matched generative managers, improving compound‑system performance by up to 0.182 points and raising FanOutQA cell F1 from 0.4778 to 0.5524.