arXiv AI By Xia Hu, Brian Potetz, Chun-Ta Lu, Huanfen Yao, Leonidas Guibas, Zhicheng Wang, Howard Zhou, Pengfei Xing, Andrew Gallagher

Where MLLMs Fail and Why: Causal Task Decomposition for Capability Failure Diagnosis

Read the original on arXiv AI →

The paper introduces a causal decomposition framework to diagnose failures in multimodal large language models (MLLMs) by distinguishing between intrinsic capability deficits and cascading errors from upstream prerequisites. It defines capability metrics (NC, IC, RC) and contribution metrics (N-Score, S-Score) to evaluate tasks under different prerequisite conditions, and applies the framework to the CADET benchmark of 10 composite tasks with 46 unit tasks and 33,000 human-annotated questions. The study finds that providing correct prerequisites eliminates 54% of errors on cognitive tasks and that a single critical prerequisite can account for 84% of the improvement gained by supplying all prerequisites.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

arXiv AI
Oct 2

DeFA: Dependency-Guided Failure Attribution for LLM Agents

DeFA is a dependency-guided framework that attributes failures in large language model agents by constructing an event dependency graph and a failure propagation graph from protocol relations and semantic dependencies. It identifies violating events, traces their sources and effects, and determines the decisive error, responsible agent, and error category. The method supports long trajectories through segmentation and has shown superior accuracy on text, image, and video tasks, while its diagnostic feedback can improve agent performance on subsequent tasks.

By Bo Deng, Xinlei Zheng, Yi Wei, Kang Zhou, Chongyang Tao, Renzhao Liang, Xuanren Chen, Lifan Guo, Chi Zhang
arXiv AI
Sep 2

DiagEvo: Diagnosis-Guided Self-Evolution via Hierarchical Error Memory

DiagEvo is a self‑evolution framework that guides language‑model training by extracting recurring error causes from a solver’s own failure history and storing them in a hierarchical error‑cause memory. The system classifies causes as Active or Mastered, uses this information to balance targeted question generation with exploration, and applies double‑confidence filtering to keep only intermediate‑difficulty questions. Experiments show that DiagEvo outperforms baselines on nine benchmarks for three solvers, achieving up to 72.3% mean accuracy on five mathematical reasoning tasks.

By Xincheng Wei, Yifan Ding, Yoshua Li, Dongsheng Ma, Rongxiang Weng, Xunliang Cai, Wenjian Ding, Yao Zhang