M4Bench is a benchmark comprising 28 clinical derivation tasks across 15 families on MIMIC‑IV and eICU datasets, designed to evaluate whether AI coding agents can reproduce complex procedural specifications for electronic health record (EHR) variable derivation. The study shows that incorporating specification completeness into agent skills yields the largest performance gains, with modest additional benefits from delivery format and schema familiarity. The benchmark, along with tasks, skills, evaluation code, and sanitized evidence, is released as open source for clinician‑reviewable use.
arXiv:2509. 02594v3 Announce Type: replace-cross Abstract: Evaluating large language models (LLMs) on their ability to generate high-quality, accurate, situationally aware answers to clinical questions requires going beyond conventional benchmarks to assess how these systems behave in complex, high-stakes clinical scenarios.
By Sandhanakrishnan Ravichandran, Shivesh Kumar, Rogerio Corga Da Silva, Miguel Romano, Reinhard Berkels, Michiel van der Heijden, Olivier Fail, Valentine Emmanuel Gnanapragasam
arXiv:2610.14918v1 Announce Type: new
Abstract: Evaluating an agent's ability to strictly satisfy diverse and procedurally complex requirements is essential to determining its practical reliability....
By Zhengxi Li, Junyi Zhou, Qiyuan Zhang, Chuhan Wu, Fuyuan Lyu, Le Tian, Xiao Zhou, Chen Ma, Yufei Wang
arXiv:2606. 14239v1 Announce Type: new Abstract: Agent skills are structured procedural packages that guide frozen LLM agents in specialized workflows.
By Haowen Gao, Haoran Chen, Can Wang, Shasha Guo, Liang Pang, Zhaoyang Liu, Huawei Shen, Xueqi Cheng
SWE-Skills-Bench is a benchmark that evaluates the impact of individual agent skills on real-world software engineering tasks. It uses 98 public skills paired with specific GitHub repositories, generating 3,920 paired evaluations across four agent-model configurations. The study finds that most skills are ineffective, with only a few providing modest gains while some are harmful, raising the overall pass ratio from 76.6% to 76.9%.
By Tingxu Han, Yi Zhang, Wei Song, Chunrong Fang, Zhenyu Chen, Youcheng Sun, Lijie Hu
SkillNEO introduces a non‑elitist evolutionary framework for refining agent skills, allowing lower‑reward versions to receive further edits and preserve useful partial repairs. By separating the selection of a version to use from the selection of versions to improve, the method identifies which steps to keep and which failures to fix, even revisiting earlier versions with new evidence. Experiments on 86 SkillsBench tasks across eight domains show significant reward gains for GPT‑5.4 Mini and substantial improvements when the evolved skills are reused by stronger models such as GPT‑5.5 and GPT‑5.6 Luna.
By Enze Ma, Chunyu Miao, Yankai Chen, Jie Yang, Liancheng Fang, Kening Zheng, Hanrong Zhang, Muyao Xu, Xinle Yu, Chengze Li, Wei-Chieh Huang, Ziwei Ren, Yunhao Du, Zhixiang Shen, Xue Liu, Philip S. Yu, Zhen Wang