arXiv AI

DIANOIA: Diagnostic Decomposition and Joint Optimization for Multi-Agent Reasoning

DIANOIA introduces a diagnostic framework for multi‑agent large language model systems, decomposing reasoning gain into three measurable channels—coverage, fidelity, and synthesis. The protocol identifies bottleneck channels for a given task and implements a corresponding multi‑agent system with role‑diverse proposers, execution‑grounded verification, and iterative synthesis. Experiments on GSM8K, AIME‑2025, MBPP, and BFCL‑SP show that DIANOIA outperforms strong baselines, achieving significant token savings and accuracy gains while accurately pinpointing the critical channels.

arXiv AI
Oct 7

Verifying Coordination in Parallel Coding Agents: NP-Bench and a Scheduling Planner

The paper introduces NP‑Bench, a benchmark and a proactive scheduling planner that coordinates parallel large‑language‑model coding agents. By partitioning work scopes and ordering merges ahead of time, the planner improves clean‑integration rates from 1/9 to 9/9 and eliminates merge conflicts, outperforming both no‑coordination and reactive‑detection baselines. It also demonstrates that cross‑session memory can eliminate repeated mistakes and that routing facts to agents does not improve long‑context accuracy at scale.

By Sumanyu Muku
arXiv AI
Oct 9

A Closer Look at Agentic BBO: Benchmarking LLM Agents for Black-Box Optimization

The paper introduces AgenticBBO-Bench, a cross‑domain benchmark for evaluating large language model agents on black‑box optimization tasks such as synthetic functions, hyperparameter tuning, database tuning, chip design, and molecular design. Experiments show that agentic BBO outperforms direct LLM‑based methods in all five domains and beats the best numerical optimizers in four. The study also analyzes the impact of optimization tools, task semantics, prior knowledge, and the LLM’s role, finding that numerical tools are not consistently helpful, task semantics are broadly useful, and numerical optimizers can capitalize on agent‑generated search trajectories.

By Ming Chen, Rong-Xi Tan, Ke Xue, Yu-Jie Zhou, Taiye Lu, Zhi-Xuan Gao, Peng Xie, Zijun Shen, Chen Lu, Haopu Shang, Chao Qian
arXiv AI
Jun 12

The Illusion of Multi-Agent Advantage

arXiv:2606. 13003v1 Announce Type: new Abstract: Prevailing wisdom posits that Multi-Agent Systems (MAS) are superior to Single-Agent Systems (SAS), citing advantages like context protection, parallel processing and distributed decision-making.

By Prathyusha Jwalapuram, Hehai Lin, Chuyuan Li, Fangkai Jiao, Sudong Wang, Yifei Ming, Zixuan Ke, Chengwei Qin, Giuseppe Carenini, Shafiq Joty
arXiv AI
Oct 28

InfraDev: Can Coding Agents Develop AI Infrastructure, and How Can We Tell?

InfraDev is a new benchmark comprising 36 tasks derived from real-world pull requests to Megatron‑LM, vLLM, and verl, designed to test coding agents on infrastructure development for training, inference, and reinforcement learning across many GPUs. Each task requires agents to reimplement multi‑file changes and run correctly, with grading that includes hidden probes and a prompt exposing only the interfaces used. The benchmark reveals that current frontier models resolve only about 39% of tasks, with probes rejecting a third of submissions that pass official tests, and that structural and state errors dominate failures.

By Yangshen Deng, Jiaqi Ruan, Yeqi Huang, Nick Lau, Guangming Sheng, Luo Mai
arXiv Machine Learning
Jun 25

ASAP: Agent-System Co-Design for Wall-Clock-Centered Auto HPO Research for ML Experiments

arXiv:2606. 25207v1 Announce Type: new Abstract: Hyperparameter Optimization (HPO) is essential for maximizing machine learning model performance, and its core challenge is sample efficiency: finding strong configurations within a limited budget.

By Taicheng Guo, Haomin Zhuang, Kehan Guo, Yujun Zhou, Nitesh V. Chawla, Olaf Wiest, Xiangliang Zhang
arXiv AI
Sep 21

Scaling Discovery through Test-Time Communication

The paper demonstrates that test‑time communication among agents can significantly outperform independent parallel attempts on complex tasks. In experiments on the ARC‑AGI‑3 benchmark, a team of $k$ communicating agents matched the success rate of $4k$ independent agents, with the advantage growing as the team size increased. The study also shows that communication enables solving tasks that no single agent can solve, and that these benefits transfer to research‑oriented problems such as polyomino packing and MNIST classifier compression, where communicating agents surpassed prior best scores.

By Jongho Park, Vasilis Kontonis, Shivam Garg, Akshay Krishnamurthy, Dimitris Papailiopoulos
arXiv AI
Oct 12

Do More Agents Help? Evaluating Accuracy and Cost in LLM Multi-Agent Workflows

The paper introduces BenchAgent, a framework for evaluating multi‑agent large language model (LLM) workflows against a single‑agent baseline, BenchAgent Core. Using GPT‑4.1 across ten reasoning, coding, and tool‑use benchmarks, the study compares six multi‑agent configurations, finding that EvoAgent achieves the highest mean pooled accuracy (79.70%) while other agents lag by 1.71–8.76 percentage points. The analysis also highlights trade‑offs in token consumption and accuracy, offering guidance for selecting and optimizing LLM multi‑agent workflows.

By Yuhang Fu, Ruishan Fang, Jiaqi Shao, Huiyu Zheng, Zhengtao Zhu, Bing Luo, Zhenyu Wen, Tao Lin
arXiv AI
Oct 12

Humanize MLE Agents: Multi-Agent Session Alternation Beyond Single-Agent Limits

The paper introduces Humanize MLE Agents (HMA), a workflow that alternates two agents across sessions on a shared workspace, limiting each session to at most k experiments and resetting context between sessions. This approach, modeled as a context‑reset semi‑Markov decision process, outperforms single‑agent baselines, achieving a 78.2% any‑medal rate on 75 Kaggle MLE‑bench tasks—8.0 points higher than the best open‑source baseline and using only a quarter of its time budget. Across six heterogeneous agent pairs, HMA improves the any‑medal rate by an average of 6.4 points, and a supplementary kernel‑optimization case reaches 878 cycles, within 1.6% of the community leaderboard leader.

By Changye Li, Zijian Zhang, Muheng Li, Yiduo Pang, Tingjun Wu, Menghan Li, Yitong Liu, Hongkai Guo, Mengchi Chen, Chenyu He, Kaisheng Ma, Wei Xu, Song Han, Sihao Liu, Ligeng Zhu