← Back to all news
arXiv AI By Tingxu Han, Yi Zhang, Wei Song, Chunrong Fang, Zhenyu Chen, Youcheng Sun, Lijie Hu

SWE-Skills-Bench: Do Agent Skills Actually Help in Real-World Software Engineering?

Read the original on arXiv AI →

SWE-Skills-Bench is a benchmark that evaluates the impact of individual agent skills on real-world software engineering tasks. It uses 98 public skills paired with specific GitHub repositories, generating 3,920 paired evaluations across four agent-model configurations. The study finds that most skills are ineffective, with only a few providing modest gains while some are harmful, raising the overall pass ratio from 76.6% to 76.9%.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

Related stories

arXiv AI
Oct 9

One Skill Too Many: How Co-Installed Skills Conflict in Coding Agents

The paper investigates how co-installed coding‑agent skills can conflict, causing the model to choose a similar skill over the intended one based solely on name and description. An empirical study of 20,947 repositories uncovered 822,109 candidate skill pairs, with 312 confirmed conflicts tested across three models, revealing that nearly one in four installed skills is overridden by a similar skill. Key findings include that conflicts often involve normative skills, that the installed skill loses core functions in about one‑fifth of runs, and that the first skill read determines the outcome, suggesting benchmarks should evaluate exclusive core functions and platforms should guard the initial read.

By Chaoliang Yan, Zihao Xu, Yuekang Li, Shangzhi Xu, Yi Liu, Gelei Deng, Siqi Ma
llmsagentsbenchmarks
More like this →