arXiv Computation and Language

Signal or Noise? A Benchmark Study of Agent Skills in Web Development

arXiv AI
3d ago

One Skill Too Many: How Co-Installed Skills Conflict in Coding Agents

The paper investigates how co-installed coding‑agent skills can conflict, causing the model to choose a similar skill over the intended one based solely on name and description. An empirical study of 20,947 repositories uncovered 822,109 candidate skill pairs, with 312 confirmed conflicts tested across three models, revealing that nearly one in four installed skills is overridden by a similar skill. Key findings include that conflicts often involve normative skills, that the installed skill loses core functions in about one‑fifth of runs, and that the first skill read determines the outcome, suggesting benchmarks should evaluate exclusive core functions and platforms should guard the initial read.

By Chaoliang Yan, Zihao Xu, Yuekang Li, Shangzhi Xu, Yi Liu, Gelei Deng, Siqi Ma
arXiv AI
Sep 30

SkillGym: Training Skill-Use Agents with Automatic Verifiable Environment Generation

SkillGym is an automatic pipeline that generates verifiable environments for training skill-use agents. It crawls internet skills, filters for reproducible workflows, and uses a builder‑reviewer process to create difficulty‑controlled tasks with reference solutions and verifiers. The system builds 6.8k environments, collects 19k successful trajectories, and fine‑tunes LLMs from 2B to 122B parameters, improving performance and skill invocation rates.

By Renxi Wang, Mingshan Hee, Fajri Koto, Timothy Baldwin, Haonan Li
arXiv AI
Aug 24

Representation Affects Retrieval: A Case Study of Skill Discovery and Routing in a Multimodal Agent Harness

The paper examines how skill representations influence selection in a multimodal video agent harness called Tinycloud. It compares two types of skill representations—tool-skills and workflow-skills—and two prompt surfaces—full inlined bodies and one-line listings—across three exposure regimes. The study finds that full autoload exposure consistently selects the correct skill, while partial exposure can cause lexical competition that misroutes tasks, highlighting that in-prompt exposure is not always beneficial.

By Kevin Dela Rosa
arXiv AI
Jun 9

Skill Retrieval Augmentation for Agentic AI

arXiv:2604. 24594v3 Announce Type: replace-cross Abstract: As large language models (LLMs) evolve into agentic problem solvers, they increasingly rely on external, reusable skills to handle tasks beyond their native parametric capabilities.

By Weihang Su, Jianming Long, Qingyao Ai, Qiaozhi He, Yichen Tang, Changyue Wang, Yiteng Tu, Yingbo Wang, Yiqun Liu
arXiv AI
Sep 28

SkillFlow: Scalable and Efficient Agent Skill Retrieval System

SkillFlow is an open, multi-stage retrieval system that helps AI agents selectively load relevant skills from a large library of community-contributed SKILL.md definitions. The pipeline uses dense retrieval, two rounds of cross-encoder reranking, and LLM-based selection to balance recall and precision. Evaluations on SkillsBench and Terminal-Bench show that SkillFlow improves performance when high-quality skills are available, but retrieval alone does not help if the corpus lacks executable skills for the target domain.

By Fangzhou Li, Pagkratios Tagkopoulos, Ilias Tagkopoulos