UMAS is a parameter‑free, cost‑neutral framework for system‑level uncertainty quantification in multi‑agent large language model systems. It estimates the credibility of inter‑agent influence using each generation’s intrinsic confidence, modulates uncertainty updates at each node, and aggregates these into a system‑level estimate. Evaluations on Debate and DyLAN show UMAS outperforms state‑of‑the‑art baselines by an average of 10.54 AUROC points and improves hallucination detection and answer selection, boosting MAS accuracy by up to 12.6 points on specific tasks.
By Hanwen Li, Jinhao Duan, Xiaoshuang Shi, Yue Zhang, Tianlong Chen, Kaidi Xu, Chenxi Yuan
The paper introduces RUPA, a trajectory‑level uncertainty quantification framework for large language model agents. RUPA models an agent’s execution as a directed graph of reasoning states, tool interactions, and environment feedback, then propagates uncertainty across this graph to capture long‑range dependencies. Experiments on benchmarks such as τ‑2, Terminal‑Bench‑2, and GAIA show that RUPA outperforms existing methods, enabling earlier failure detection and more reliable agent execution.
By Zhengzhao Ma. Boxi Cao, Yaojie Lu, Hongyu Lin, Xianpei Han, Le Sun
The paper introduces Interaction-Effect Propagation (IE-Prop), a model that estimates error risk in large language model (LLM) based multi‑agent systems by conditioning on whether upstream agents are correct or incorrect. IE‑Prop learns separate effects for each sender‑receiver interaction, combines them using the sender’s estimated error risk, and recursively propagates these risks through the execution graph. Experiments on three architectures and benchmarks show that IE‑Prop improves final‑node AUROC by about 0.05 points over baselines and that multi‑step risk tracking adds an additional 0.03 AUROC in sequential systems.
By Shreyan Ganguly, Souradeep Chattopadhyay, Qisai Liu, Soumik Sarkar
arXiv:2611.07140v1 Announce Type: cross
Abstract: Teams increasingly build software from cooperating large language model (LLM) agents, often assuming more agents mean more reliability. Yet the usual...
By Burkay Tun\c{c}t\"urk, Koorosh Aslansefat, Yiannis Papadopoulos, Bao Nguyen
The paper introduces EMFA, a method for pinpointing the decisive error in large‑language‑model‑based multi‑agent systems. EMFA builds a structured representation of a failed execution, models how errors cascade and loop, and then screens candidates with propagation awareness before verifying them counterfactually. On the Who&When benchmark, EMFA outperforms prior methods, improving step‑level attribution accuracy by 3.45 and 4.40 percentage points on two subsets.
The paper introduces EMFA, a method for attributing failures in large language model-based multi-agent systems by identifying the decisive error—an agent–step pair whose correction would restore correct execution. EMFA builds a structured representation of the failed trajectory, models error propagation and persistent loops, and applies propagation-aware screening with counterfactual verification to pinpoint the decisive error. On the Who&When benchmark, EMFA outperforms prior methods, improving step-level attribution accuracy by 3.45 and 4.40 percentage points on two subsets.
By Jiaqi Liao, Yuanzhao Zhai, Huanxi Liu, Xu Zhang, Zheming Zhuang, Dawei Feng, Bo Ding, Huaimin Wang
arXiv:2607. 02186v1 Announce Type: new Abstract: Software development is a complex task that demands cooperation among agents with diverse roles.
By Temitayo Olamilekan Ogunsusi, Lijun Qian, Xishuang Dong
arXiv:2607. 07989v1 Announce Type: cross Abstract: Large language model (LLM) based multi-agent systems enable complex problem solving through coordinated reasoning and action, but their distributed structure also introduces new challenges in diagnosing system-level failures.
By Yufei Xia, Anjun Gao, Yueyang Quan, Zhuqing Liu, Minghong Fang
arXiv:2610.15459v1 Announce Type: cross
Abstract: LLM-based multi-agent systems (LLM-MAS) use specialized agents that exchange messages to accomplish complex tasks. Their reliance on networked protoc...
By Jinhao Zhou, Xiaoyu Zhang, Jiongchi Yu, Fahao Chen, Jun Wu
arXiv:2607. 10811v1 Announce Type: cross Abstract: AI engineering is shifting from passive text generation by large language models (LLMs) to agent-driven task execution, creating new reliability challenges for long-horizon tasks under resource constraints and environmental uncertainty.
By Kai Yu, Lu Chen, Hanqi Li
GraphACM introduces a graph-based framework to measure fine-grained action consistency in large language model agents. By merging equivalent states into a directed, action-labeled graph across multiple task rollouts, it enables comparisons of actions at shared decision points and defines node-level and global consistency metrics. Experiments on ALFWorld, AppWorld, and WebShop show that the global consistency score predicts task success with an AUROC of 0.811, outperforming prior consistency baselines and verbalized confidence.
By Yaxin Ma, Wenbo Zhang, Satya Krishna Pothapragada, Alina Zare
Behavioral consistency across repeated executions provides an informative signal for assessing the reliability of large language model (LLM) agents. However, existing approaches typically summarize co...