arXiv AI
Aug 13
Agent Skills Can Be Harmful: An Empirical Study of Skill-Induced Failures in LLM Agents
arXiv:2608. 11888v1 Announce Type: new Abstract: Agent skills are the de facto mechanism for extending LLM agents with reusable guidance.
Where MLLMs Fail and Why: Causal Task Decomposition for Capability Failure Diagnosis
The paper introduces a causal decomposition framework to diagnose failures in multimodal large language models (MLLMs) by distinguishing between intrinsic capability deficits and cascading errors from upstream prerequisites. It defines capability metrics (NC, IC, RC) and contribution metrics (N-Score, S-Score) to evaluate tasks under different prerequisite conditions, and applies the framework to the CADET benchmark of 10 composite tasks with 46 unit tasks and 33,000 human-annotated questions. The study finds that providing correct prerequisites eliminates 54% of errors on cognitive tasks and that a single critical prerequisite can account for 84% of the improvement gained by supplying all prerequisites.
Related stories
arXiv AI
Jun 16
Not All Skills Help: Measuring and Repairing Agent Knowledge
arXiv:2606. 15390v1 Announce Type: cross Abstract: LLM agents can improve without weight updates by accumulating natural-language skills from experience, but current systems entrust every decision about which skills to keep and how to apply them to LLM judgment alone.
arXiv AI
Oct 2
DeFA: Dependency-Guided Failure Attribution for LLM Agents
DeFA is a dependency-guided framework that attributes failures in large language model agents by constructing an event dependency graph and a failure propagation graph from protocol relations and semantic dependencies. It identifies violating events, traces their sources and effects, and determines the decisive error, responsible agent, and error category. The method supports long trajectories through segmentation and has shown superior accuracy on text, image, and video tasks, while its diagnostic feedback can improve agent performance on subsequent tasks.
Hugging Face Trending Papers
Aug 12
Agent Skills Can Be Harmful: An Empirical Study of Skill-Induced Failures in LLM Agents
Agent skills are the de facto mechanism for extending LLM agents with reusable guidance. A skill can shape the agent's task execution, including planning, tool use, problem-solving, and validation.
Hugging Face Trending Papers
Oct 20
Foreword: Pre-Generation Prediction and Characterisation of LLM Test Failures using the Jacobian Lens
Testing LLM-based software is expensive: each input needs several sampled responses, each judged by a costly oracle. Further, every prompt or model change calls for re-testing. Pre-generation adequacy...
arXiv AI
Sep 2
DiagEvo: Diagnosis-Guided Self-Evolution via Hierarchical Error Memory
DiagEvo is a self‑evolution framework that guides language‑model training by extracting recurring error causes from a solver’s own failure history and storing them in a hierarchical error‑cause memory. The system classifies causes as Active or Mastered, uses this information to balance targeted question generation with exploration, and applies double‑confidence filtering to keep only intermediate‑difficulty questions. Experiments show that DiagEvo outperforms baselines on nine benchmarks for three solvers, achieving up to 72.3% mean accuracy on five mathematical reasoning tasks.
arXiv Machine Learning
Oct 21
Foreword: Pre-Generation Prediction and Characterisation of LLM Test Failures using the Jacobian Lens
arXiv:2610.28187v1 Announce Type: cross Abstract: Testing LLM-based software is expensive: each input needs several sampled responses, each judged by a costly oracle. Further, every prompt or model c...
arXiv AI
4d ago
Using profiles of cognitive capability to assess AI suitability for workplace tasks
The paper proposes a method for assessing AI suitability for workplace tasks by profiling both AI agents and tasks along shared cognitive capability dimensions. It uses benchmark performance to infer agent capabilities and expert input to weight task requirements, allowing independent updates as models and roles evolve. The authors validate the approach on synthetic agents, six AI systems, and 410 employees across six domains, showing that AI systems vary more by cognitive dimension than by model family, while workplace tasks share a common cognitive core.
arXiv AI
Oct 7
ChartBmkAgent: Harness-Governed Multi-Agent Construction of Chart QA Benchmarks from Sparse Error-Taxonomy Specifications
arXiv:2610.08106v1 Announce Type: new Abstract: Multimodal large language models (MLLMs) advance rapidly, while conventional benchmark development lags behind, delaying investigation of newly observe...
arXiv Machine Learning
Jul 30
Same Evidence, Different Target: Decoding How Diagnostic Evidence Bears on Causal Questions from Language-Model States
arXiv:2607. 26929v1 Announce Type: cross Abstract: The same diagnostic result can support or challenge one causal claim yet fail to address another when the claims concern different populations, outcomes, estimands, pathways, or identifying assumptions.
arXiv AI
Sep 7
DCFA: Dual-view Causal-inspired Attribution for Failure Reasoning in LLM-based Multi-agent Systems
The paper introduces DCFA, a training‑free framework for attributing failures in large language model‑based multi‑agent systems. DCFA uses a global module to build causal‑inspired dependency graphs from system traces, pinpointing the earliest decisive error, and a local module that refines this attribution through counterfactual reasoning. Experiments on the Who&When benchmark across six LLMs demonstrate that DCFA improves step‑level accuracy by up to 8.27% over existing baselines.
arXiv AI
Aug 25
GIM: Evaluating models via tasks that integrate multiple cognitive domains
The paper introduces the Grounded Integration Measure (GIM), a benchmark of 820 expert‑authored problems designed to test models on tasks that integrate multiple cognitive operations such as constraint satisfaction, state tracking, epistemic vigilance, and audience calibration. GIM emphasizes realistic, broadly accessible knowledge rather than specialized expertise, and uses a judge‑aware 2‑parameter logistic IRT model to produce robust ability estimates across 53 model‑thinking‑level configurations. The authors provide a comprehensive leaderboard of 22 models and 47 test configurations, and conduct an extensive study on how test‑time compute affects model capability, finding that configuration choices like thinking budget and quantization can be as influential as model selection itself. whyItMatters":"By focusing on integration of multiple cognitive domains, GIM offers a more realistic assessment of model reasoning capabilities than benchmarks that either overemphasize memorization or abstract reasoning alone."