GraphACM introduces a graph-based framework to measure fine-grained action consistency in large language model agents. By merging equivalent states into a directed, action-labeled graph across multiple task rollouts, it enables comparisons of actions at shared decision points and defines node-level and global consistency metrics. Experiments on ALFWorld, AppWorld, and WebShop show that the global consistency score predicts task success with an AUROC of 0.811, outperforming prior consistency baselines and verbalized confidence.
By Yaxin Ma, Wenbo Zhang, Satya Krishna Pothapragada, Alina Zare
UMAS is a parameter‑free, cost‑neutral framework for system‑level uncertainty quantification in multi‑agent large language model systems. It estimates the credibility of inter‑agent influence using each generation’s intrinsic confidence, modulates uncertainty updates at each node, and aggregates these into a system‑level estimate. Evaluations on Debate and DyLAN show UMAS outperforms state‑of‑the‑art baselines by an average of 10.54 AUROC points and improves hallucination detection and answer selection, boosting MAS accuracy by up to 12.6 points on specific tasks.
By Hanwen Li, Jinhao Duan, Xiaoshuang Shi, Yue Zhang, Tianlong Chen, Kaidi Xu, Chenxi Yuan
arXiv:2608. 13598v1 Announce Type: new Abstract: Agent evaluation relies almost entirely on outcome metrics such as success rate, which capture whether an agent succeeds but not how consistently it behaves.
By Amritesh Banerjee, Pranil Raichura
The paper introduces RUPA, a trajectory‑level uncertainty quantification framework for large language model agents. RUPA models an agent’s execution as a directed graph of reasoning states, tool interactions, and environment feedback, then propagates uncertainty across this graph to capture long‑range dependencies. Experiments on benchmarks such as τ‑2, Terminal‑Bench‑2, and GAIA show that RUPA outperforms existing methods, enabling earlier failure detection and more reliable agent execution.
By Zhengzhao Ma. Boxi Cao, Yaojie Lu, Hongyu Lin, Xianpei Han, Le Sun
arXiv:2607. 16868v1 Announce Type: new Abstract: Large Language Models (LLMs) often produce confidently stated yet unreliable outputs, posing critical challenges for deployment in safety-sensitive applications.
By Yanni Dong, Minghua Liu, Meiling Zhu, Xiaowei Huang, Lijun Zhang
arXiv:2607. 13884v1 Announce Type: new Abstract: Large Language Model (LLM) agents have shown remarkable capabilities in autonomous decision-making by generating sequential trajectories of states, actions, and observations.
By Wenjun Wang, Yuchen Fang, Fengrui Liu, Zibo Liang, Kai Zheng
arXiv:2608.29971v1 Announce Type: new
Abstract: As agentic systems evolve into complex multi agent orchestration workflows, there is a growing and critical need for systematic frameworks that measure...
By Ram Kulathumani, Regunathan Radhakrishnan, Anupam Tripathi, Xiangbo Mao, Roshanak Omrani, Keshav Somani, Shwet Kamal Mishra, Shayna Lurya
The paper introduces FRESH, a Failure-aware Retrieval framework that uses Experience-Structured Heterogeneous graphs to transform past successes and failures into structured external memory for tool‑using agents. By explicitly modeling dependencies among tasks, actions, errors, repairs, and execution conditions, FRESH enables frozen language models to reuse reliable strategies, avoid recurring failures, and make safer decisions in stateful tool interactions. Experiments on τ‑Bench and AppWorld with multiple open‑source models demonstrate that FRESH consistently improves task success and tool‑use reliability compared to no‑memory agents and other memory‑based baselines.
By Jiaxing Li, Lei Song, Rui Dong, Youyong Kong
The paper introduces Recursive Semantic Divergence (RSD), an unsupervised, trajectory‑level consistency metric for large language model agents that use tools. RSD recursively measures expected cumulative semantic disagreement between two independent rollouts from the same state, using natural language inference to detect logical inconsistency. When applied as a reward signal during fine‑tuning, RSD significantly reduces variability in task progress across benchmarks and model scales while matching supervised fine‑tuning performance without needing ground‑truth labels.
By Harshavardhan Abichandani, Penny Chong, Atin Ghosh, Daniel Dahlmeier
arXiv:2504.18346v4 Announce Type: replace-cross
Abstract: Large Language Models (LLMs) have been transformative across many domains. However, hallucination, i.e., confidently outputting incorrect inf...
By Toghrul Abbasli, Kentaroh Toyoda, Yuan Wang, Leon Witt, Muhammad Asif Ali, Yukai Miao, Dan Li, Qingsong Wei
The paper introduces Experience Memory Graph (EMG), a framework that transforms agent failure recovery into a graph matching problem. By converting failed and successful trajectories into directed action decision graphs, EMG extracts common subgraphs and edit paths that reveal explicit correction strategies, storing them in a memory graph with intra‑task nodes and cross‑task edges. During testing, EMG retrieves relevant insights to guide the agent in a single attempt, improving performance on ALFWorld and ScienceWorld without extra reflection rounds.
By Wenjun Wang, Yuchen Fang, Fengrui Liu, Zibo Liang, Kai Zheng
PropUQ-MAS is a framework for uncertainty quantification in large language model (LLM) multi‑agent systems that models the system as a communication‑structured graph. It estimates the reliability of each step by combining local uncertainty with uncertainty inherited from upstream messages, addressing the risk of error propagation in inter‑agent communication. Experiments show consistent improvements in UQ metrics, with average gains of +6.10% in AUROC and +47.58% in PRR.
By Yaokun Liu, Yifan Liu, Daniel Yue Zhang, Ruichen Yao, Zelin Li, Dong Wang