arXiv Computation and Language

RESKILL: Explicit Failure Attribution and Structured Repair for Interactive Language Agents

arXiv AI
Sep 1

Understanding Automated Program Repair Agents Through the Lens of Traceability: An Empirical Study

The paper presents a systematic analysis of five state‑of‑the‑art automated program repair agents, tracing their decision‑making across 500 real‑world repair tasks. It finds that while the agents perform well on simple fixes, they struggle with logic‑intensive bugs, often producing verbose, overfitted patches that pass tests without addressing root causes. Key bottlenecks identified include poor test generation, limited regression test selection, and reliance on primitive tooling without access to debuggers or advanced program analysis tools.

By Ira Ceka, Hailie Mitchell, Saurabh Pujar, Luca Buratti, Shyam Ramji, Junfeng Yang, Gail Kaiser, Baishakhi Ray
arXiv AI
Sep 18

SkillAA: Attribution-Guided Skill-Graph Updating with Targeted Validation and Rollback

SkillAA (Skill Abductive Attribution) is a structured skill‑optimization framework designed for frozen language models. It represents skill applicability, execution, and composition in a unified graph, enabling skill selection, attribution‑guided repair, and update validation within the same structure. By contrasting successful and failed executions, SkillAA routes candidate repairs to specific graph objects, updates only the selected local structure, and uses Local and Big Gates to screen candidate changes before commitment, achieving high performance on SearchQA, LiveMath, and DocVQA.

By Ziqiao Shang, Ling-Yue Ge, Lan-Zhe Guo
arXiv AI
Jun 9

REFLECT: Intervention-Supported Error Attribution for Silent Failures in LLM Agent Traces

arXiv:2606. 09071v1 Announce Type: new Abstract: Large language model (LLM) agents now solve complex tasks through long plan-and-execution traces, yet the ability to locate errors in a completed traces still lags far behind, especially in the \emph{silent failure} regime.

By Xiaofeng Lin, Yingxu Wang, Tung Sum Thomas Kwok, Daniel Guo, Sahil Arun Nale, Charles Fleming, Guang Cheng
arXiv AI
3d ago

CART: Closed-Loop Adaptive Red Teaming for Large Language Models

CART (Closed‑Loop Adaptive Red Teaming) is a framework that iteratively uses results from red‑teaming tests to guide subsequent probes, thereby expanding risk coverage and maintaining diversity. It separates the roles of Challenger (test generator), Target (model or agent under test), and Judge (result evaluator), enabling independent study of each component. Across multiple evaluation families, CART uncovers more failures and higher risk than static prompt replay, demonstrating that adaptive, continuous testing reveals weaknesses that fixed‑prompt methods miss.

By Dongdong Zhang, Tengchao Lv, Yilin Jia, Yuzhong Zhao, Yupan Huang, Wenshan Wu, Xiangyang Zhou, Shaohan Huang, Nan Yang, Li Dong, Lei Cui, Furu Wei
arXiv AI
Aug 7

SearchAuditor: Auditing and Attributing Failures in Long-Horizon Search Agents

arXiv:2608. 05212v1 Announce Type: new Abstract: Deep search agents tackle challenging questions through long-horizon web interactions, a process that is both complex and fragile: small reasoning errors may propagate through long, noisy trajectories into fluent but incorrect answers.

By Zhixiang Liang, Yifei Liu, Yidan Huang, Haozhe Zhao, Beichen Huang, Jiaqi Wang, Nan Duan, Qiong Cao