arXiv AI
Passing Tests, Missing Semantics: Evidence of Interpretive Lock-In in Plan-Based DataFrame Code Generation
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv Computation and Language
Oct 23
What the Specification Did Not Say: Specification-Oracle Gaps in Test-Graded Coding-Agent Benchmarks
The paper identifies a "specification-oracle gap" in coding‑agent benchmarks, where tests rely on details (constants, module paths, exact messages) that are not stated in the natural‑language task description. The authors formalise this gap, develop a static audit tool that classifies such dependencies across five languages, and demonstrate that many benchmarks (e.g., SWE‑Milestone) contain tasks whose tests depend on unstated information, leading to higher failure rates for agents that do not reproduce upstream implementation details. The study also shows that the prevalence of these gaps varies with benchmark construction and that removing affected tests does not alter leaderboard rankings but shifts scores within repositories, favoring agents that replicate upstream names. whyItMatters":"The findings reveal that benchmark tests can inadvertently measure an agent’s ability to mimic upstream implementation details rather than its true problem‑solving skills, potentially skewing evaluations and comparisons across agents."
arXiv AI
Jul 21
AEVAL: From Anecdotal to Deterministic Testing for Agentic Skill Workflows
arXiv:2607. 16345v1 Announce Type: cross Abstract: Modern agentic systems increasingly rely on skills: installable packages of natural language and code that teach an LLM agent to perform a domain task.
arXiv AI
Oct 16
Learning When to Withdraw: Curriculum Coverage and Rule-Specific Failures in Fine-Tuned Language Agents
The paper investigates how fine‑tuned language agents can selectively withdraw actions when evidence changes or authority is revoked. Using a synthetic NAQD‑Env environment, the authors compare an original curriculum with an enriched one across multiple system‑prompt framings and evaluate four LoRA checkpoints of Qwen2.5‑3B‑Instruct against instruction‑tuned baselines. Results show that fine‑tuned checkpoints achieve higher decision agreement and withdrawal recall, reduce incorrect withdrawals, and narrow prohibited combinations, though they also omit valid event reports and exhibit mixed preservation of authorized actions.
arXiv AI
Aug 28
Evaluating human and LLM screening workflows in a conceptually complex scoping review: Recall--workload trade-offs and run-to-run consistency
The study compared human and large language model (LLM) workflows for title‑and‑abstract screening in a complex scoping review. Human reviewers and two GPT‑5.4 file‑batch runs retained 42.2‑45.0% of records with 82.3‑82.9% recall, while Gemini 3.1 achieved the highest recall (83.9%) but retained 56.7% of records. Identical GPT‑5.4 runs showed 91.7% agreement yet differed on 94 records, including 29 verified eligible ones.
arXiv AI
Aug 13
Harness-IF: Evaluating Instruction Following Across Instruction Surfaces in Coding Agents
arXiv:2608. 11727v1 Announce Type: new Abstract: When a coding agent obeys a rule, it may simply have been going to do that anyway.
arXiv AI
Oct 1
NAQD Env: A benchmark for selective withdrawal in language agents
The paper introduces NAQD‑Env, a synthetic benchmark designed to test language agents’ ability to selectively withdraw and resume actions when new evidence, permissions, or stop instructions arise. It evaluates models against a deterministic reference policy across eleven dependency families, measuring policy agreement, task value, withdrawal, resumption, and event reporting. Experiments on 350 scenarios show low withdrawal recall, no valid resumption, and limited policy alignment, highlighting the need to treat selective withdrawal as a distinct reliability component.