The paper introduces a two-stage diagnostic evaluation to assess the structural reliability of reasoning graphs used in large language models for tasks like multi-hop question answering. Stage 1 examines how graph reliability depends on topology, prompt design, and model family, while Stage 2 evaluates the impact of graph errors on uncertainty estimation. Findings show that graph-based computation is vulnerable to upstream errors, offers limited benefit as prompt context alone, but improves confidence-based answer discrimination and enables node-level diagnostics.
By Xinyu Liu, Dongqi Fu, Jun Wu
The paper introduces FRESH, a Failure-aware Retrieval framework that uses Experience-Structured Heterogeneous graphs to transform past successes and failures into structured external memory for tool‑using agents. By explicitly modeling dependencies among tasks, actions, errors, repairs, and execution conditions, FRESH enables frozen language models to reuse reliable strategies, avoid recurring failures, and make safer decisions in stateful tool interactions. Experiments on τ‑Bench and AppWorld with multiple open‑source models demonstrate that FRESH consistently improves task success and tool‑use reliability compared to no‑memory agents and other memory‑based baselines.
By Jiaxing Li, Lei Song, Rui Dong, Youyong Kong
SkillAA (Skill Abductive Attribution) is a structured skill‑optimization framework designed for frozen language models. It represents skill applicability, execution, and composition in a unified graph, enabling skill selection, attribution‑guided repair, and update validation within the same structure. By contrasting successful and failed executions, SkillAA routes candidate repairs to specific graph objects, updates only the selected local structure, and uses Local and Big Gates to screen candidate changes before commitment, achieving high performance on SearchQA, LiveMath, and DocVQA.
By Ziqiao Shang, Ling-Yue Ge, Lan-Zhe Guo
arXiv:2608. 14109v1 Announce Type: new Abstract: Autonomous LLM agents are increasingly deployed in complex real-world workflows, yet they remain vulnerable to runtime behavioral drift, a silent deviation from the original task that can lead to irreversible side effects on external systems.
By Ismail El Hamraoui, Sagar Jose, Nicolas Bureau, Robert Plana
DynSTEER is a dynamic stage‑wise trajectory evaluation framework designed for large language model agents performing long‑horizon tasks. It segments rollouts into stages anchored by key actions, uses a path‑tolerant milestone graph to accommodate diverse valid strategies, and adaptively routes queries to multi‑tier judges while halting unrecoverable executions early. Experiments show it improves evaluation discriminability by 85.2% over native methods, separates all model pairs with statistical significance, and saves 34.51% of execution steps on failed rollouts.
By Zhichao Shi, Wenjie Zhang, Xuhui Jiang, Xiaojun Wu, Cehao Yang, Chengjin Xu, Jian Guo, Yuanzhuo Wang
GraphDecisionBench is a benchmark designed to evaluate System One models—such as Jev—that make decisions directly from supplied options on graph-related tasks. The benchmark is model‑independent and incorporates structural task profiles, matched graph‑text input contrasts, and heuristic‑proposal controls to diagnose graph decision performance. Evaluations across fourteen model‑interface configurations reveal that accurate adjacency recognition does not guarantee broader structural correctness, joint graph‑text input does not consistently improve prediction, and feasible construction does not ensure high solution quality.
By Xianliang Yang, Yapu Zhang, Li Zhao
arXiv:2604.10800v2 Announce Type: replace-cross
Abstract: Learned classifiers deployed in agentic pipelines face a fundamental reliability problem: predictions are probabilistic inferences, not verif...
By Jugal Gajjar
arXiv:2606. 27780v1 Announce Type: new Abstract: World models are often used for planning by rolling learned dynamics forward.
By Xinyuan Song, Zekun Cai
GraphDecide is a model‑independent benchmark designed to evaluate System One models—such as Jev—that make decisions directly from supplied options on graph‑related tasks. The benchmark combines structural task profiles, matched graph‑text input contrasts, and heuristic‑proposal controls to diagnose graph decision performance. In testing fourteen model‑interface configurations, GraphDecide shows that accurate adjacency recognition does not guarantee broader structural correctness, joint graph‑text input does not consistently improve prediction, and feasible construction does not ensure high solution quality.
arXiv:2608.29971v1 Announce Type: new
Abstract: As agentic systems evolve into complex multi agent orchestration workflows, there is a growing and critical need for systematic frameworks that measure...
By Ram Kulathumani, Regunathan Radhakrishnan, Anupam Tripathi, Xiangbo Mao, Roshanak Omrani, Keshav Somani, Shwet Kamal Mishra, Shayna Lurya
arXiv:2606. 27806v1 Announce Type: new Abstract: World models for language agents come in two useful forms.
By Xinyuan Song, Zekun Cai
arXiv:2608. 08700v1 Announce Type: new Abstract: Reliable evaluation of tool routing is critical as Large Language Models increasingly operate as autonomous agents.
By Dongjie Xu, Julius, Hanchi Dong, Minghua Tang, Yuxuan Sun, Ziwei Nie, Zicheng Liu, Dujun Qing, Jiajie Xu