Behavioral consistency across repeated executions provides an informative signal for assessing the reliability of large language model (LLM) agents. However, existing approaches typically summarize co...
The paper introduces RUPA, a trajectory‑level uncertainty quantification framework for large language model agents. RUPA models an agent’s execution as a directed graph of reasoning states, tool interactions, and environment feedback, then propagates uncertainty across this graph to capture long‑range dependencies. Experiments on benchmarks such as τ‑2, Terminal‑Bench‑2, and GAIA show that RUPA outperforms existing methods, enabling earlier failure detection and more reliable agent execution.
By Zhengzhao Ma. Boxi Cao, Yaojie Lu, Hongyu Lin, Xianpei Han, Le Sun
UMAS is a parameter‑free, cost‑neutral framework for system‑level uncertainty quantification in multi‑agent large language model systems. It estimates the credibility of inter‑agent influence using each generation’s intrinsic confidence, modulates uncertainty updates at each node, and aggregates these into a system‑level estimate. Evaluations on Debate and DyLAN show UMAS outperforms state‑of‑the‑art baselines by an average of 10.54 AUROC points and improves hallucination detection and answer selection, boosting MAS accuracy by up to 12.6 points on specific tasks.
By Hanwen Li, Jinhao Duan, Xiaoshuang Shi, Yue Zhang, Tianlong Chen, Kaidi Xu, Chenxi Yuan
arXiv:2608. 13598v1 Announce Type: new Abstract: Agent evaluation relies almost entirely on outcome metrics such as success rate, which capture whether an agent succeeds but not how consistently it behaves.
By Amritesh Banerjee, Pranil Raichura
arXiv:2607. 13884v1 Announce Type: new Abstract: Large Language Model (LLM) agents have shown remarkable capabilities in autonomous decision-making by generating sequential trajectories of states, actions, and observations.
By Wenjun Wang, Yuchen Fang, Fengrui Liu, Zibo Liang, Kai Zheng
arXiv:2608.29971v1 Announce Type: new
Abstract: As agentic systems evolve into complex multi agent orchestration workflows, there is a growing and critical need for systematic frameworks that measure...
By Ram Kulathumani, Regunathan Radhakrishnan, Anupam Tripathi, Xiangbo Mao, Roshanak Omrani, Keshav Somani, Shwet Kamal Mishra, Shayna Lurya