The paper identifies a "specification-oracle gap" in coding‑agent benchmarks, where tests rely on details (constants, module paths, exact messages) that are not stated in the natural‑language task description. The authors formalise this gap, develop a static audit tool that classifies such dependencies across five languages, and demonstrate that many benchmarks (e.g., SWE‑Milestone) contain tasks whose tests depend on unstated information, leading to higher failure rates for agents that do not reproduce upstream implementation details. The study also shows that the prevalence of these gaps varies with benchmark construction and that removing affected tests does not alter leaderboard rankings but shifts scores within repositories, favoring agents that replicate upstream names.
whyItMatters":"The findings reveal that benchmark tests can inadvertently measure an agent’s ability to mimic upstream implementation details rather than its true problem‑solving skills, potentially skewing evaluations and comparisons across agents."
By Fei Ding
arXiv:2607. 16345v1 Announce Type: cross Abstract: Modern agentic systems increasingly rely on skills: installable packages of natural language and code that teach an LLM agent to perform a domain task.
By Tejas Singh Anand, Yuet Ying Christina Wang, Wanting Jiang, Steve Masson, Tian Zheng, Bingjie Zhou
The paper investigates how fine‑tuned language agents can selectively withdraw actions when evidence changes or authority is revoked. Using a synthetic NAQD‑Env environment, the authors compare an original curriculum with an enriched one across multiple system‑prompt framings and evaluate four LoRA checkpoints of Qwen2.5‑3B‑Instruct against instruction‑tuned baselines. Results show that fine‑tuned checkpoints achieve higher decision agreement and withdrawal recall, reduce incorrect withdrawals, and narrow prohibited combinations, though they also omit valid event reports and exhibit mixed preservation of authorized actions.
By Mohamed Abouzahra
The study compared human and large language model (LLM) workflows for title‑and‑abstract screening in a complex scoping review. Human reviewers and two GPT‑5.4 file‑batch runs retained 42.2‑45.0% of records with 82.3‑82.9% recall, while Gemini 3.1 achieved the highest recall (83.9%) but retained 56.7% of records. Identical GPT‑5.4 runs showed 91.7% agreement yet differed on 94 records, including 29 verified eligible ones.
By Nikol Figalov\'a, Lynn Huestegge, Anne B\"ockler-Raettig
arXiv:2608. 11727v1 Announce Type: new Abstract: When a coding agent obeys a rule, it may simply have been going to do that anyway.
By Zining Huang, Haoran Que, Hong Zeng, Ge Zhang, Zuo Wang, Jin Chen, Haodong Wang, Zhongfei Hou, Changxin Pu, Shen Yan, Wenhao Huang
The paper introduces NAQD‑Env, a synthetic benchmark designed to test language agents’ ability to selectively withdraw and resume actions when new evidence, permissions, or stop instructions arise. It evaluates models against a deterministic reference policy across eleven dependency families, measuring policy agreement, task value, withdrawal, resumption, and event reporting. Experiments on 350 scenarios show low withdrawal recall, no valid resumption, and limited policy alignment, highlighting the need to treat selective withdrawal as a distinct reliability component.
By Mohamed Abouzahra
arXiv:2606. 30573v1 Announce Type: new Abstract: We introduce SWE-Interact, a new testbed for evaluating coding agents on multi-turn, interactive, user-driven software engineering tasks.
By Mohit Raghavendra, Anisha Gunjal, Aakash Sabharwal, Yunzhong He
The paper investigates how language‑model judges can make version‑dependent errors when evaluating upgraded agents. Using 35 public coding‑agent submissions, two customer‑service agents, and over a thousand expert‑labeled trajectories, the authors show that fixed judges often reject task‑conditioned error invariance and can incorrectly approve failed patches, especially as agent capability increases. Paired audits of current outputs reduce interval width only marginally, and the study concludes that independent human patch review is still necessary.
By Jiapeng Li
The paper presents a cross‑provider review contract for coding agents, detailing resource separation, bounded execution, and explicit failure handling. In a controlled pilot of 20 paired development turns, 8 reviews yielded material findings, and a boundary‑condition scan uncovered and repaired several defects, including false successes and cancellation issues. Subsequent CLI probes and a preregistered shadow study identified additional defects related to reviewer exit status and tool usage, leading to an audit cohort and ongoing data collection.
By Bowen Xu, Boyu Chen
arXiv:2608. 19475v1 Announce Type: cross Abstract: This work introduces a formal semantic-block model for specifications and an execution-judged benchmark for evaluating specification quality independently of model capability.
By Oleg Grynets, Dmytro Kostetskyi, Vasyl Lyashkevych
The study investigates how deictic ambiguity—specifically the shifting reference of expressions like "previous"—affects Draft‑Verify‑Revise pipelines that use multiple large language models (LLMs). Using a synthetic dataset of 10 base examples and 21 reasoning‑effort configurations, six LLMs were evaluated for their ability to correctly resolve the ambiguous expression across the draft, verify, and revise stages. Results show wide variance in balanced accuracy, with GPT‑5.2 improving from 0.156 to 0.942 with increased reasoning effort, while Gemini 3 Pro consistently achieved high accuracy above 0.94 even at low reasoning effort, and meta‑evaluators often relied on surface cues when making errors.
By Obinna I. Ekekezie
M4Bench is a benchmark comprising 28 clinical derivation tasks across 15 families on MIMIC‑IV and eICU datasets, designed to evaluate whether AI coding agents can reproduce complex procedural specifications for electronic health record (EHR) variable derivation. The study shows that incorporating specification completeness into agent skills yields the largest performance gains, with modest additional benefits from delivery format and schema familiarity. The benchmark, along with tasks, skills, evaluation code, and sanitized evidence, is released as open source for clinician‑reviewable use.