BTS-AgentBench is a deterministic, replayable pipeline that transforms read‑only telemetry logs from industrial sites into executable multi‑turn agent benchmarks. The method normalizes BTS metadata and raw histories into a read‑only tool store, compiles static tasks with tool‑derived gold answers and evidence, and lifts retained tasks into typed, bounded operator‑facing episodes. The 532‑row release includes clarifications, goal revision, timestamp policy, quality‑gated reporting, and evidence attribution, and the construction process reproduces all released train/dev/test artifacts exactly.
arXiv:2605. 11030v2 Announce Type: replace-cross Abstract: Closed-loop tool-using agents are increasingly evaluated in executable web, code, and micro-task environments, but benchmark reports often conflate workloads, action-generating drivers, and the evidence admitted for systems-facing claims.
arXiv:2607. 08010v1 Announce Type: cross Abstract: Production LLM agents often waste latency and reliability by regenerating code for the same procedural steps on every request.
arXiv:2609.38201v1 Announce Type: new Abstract: Long-running tools can dominate coding-agent latency: compilers, test suites, and repository commands take seconds to minutes while the agent idles. Th...
The paper introduces Soei, a structured observability framework for AI coding agents, addressing the need to track what work was done, by whom, and whether it is finished. It evaluates an existing tool, Agenttrail, revealing inconsistencies and security gaps, and presents empirical findings from 45 real coding-agent sessions, highlighting issues such as unverified code edits and credential exposure. Soei comprises an event schema, hash‑chained ledger, PROV‑aligned graph, and verification layer, achieving efficient storage, fast verification, and tamper detection on the same dataset.
Production LLM agents often waste latency and reliability by regenerating code for the same procedural steps on every request. We replace this inference-time coding loop with an agentic tool-making pipeline that compiles repeated SOP steps into validated, versioned tools before deployment.
Agent Harness Awareness addresses the gap between tool descriptions and real‑time operational suitability by having the host provide a verified, up‑to‑date state snapshot before an agent dispatches an action. In controlled benchmarks, this approach cut observable policy regret by nearly 97 % and boosted single‑dispatch completion from 69.4 % to 95.8 %. In live, fault‑injected tests, verified state eliminated excess operational cost and achieved 100 % completion across all model‑by‑fault scenarios.
The paper discusses the problem of benchmark maintenance debt in evaluating coding agents, where benchmarks become outdated as models improve and infrastructure changes. It introduces a continuous‑validation workflow that incorporates community feedback, automated checks, and human repairs, applied to Terminal‑Bench 2.0. The audit uncovered issues in about 30% of the benchmark, leading to a new version (Terminal‑Bench 2.1) that improved accuracy by over 10 percentage points and altered leaderboard positions, while also highlighting the importance of validating resource budgets alongside task artifacts.
The paper introduces rebuild‑dossier, an open‑source tool that locks an application’s real interface before code is written and enforces one‑test‑at‑a‑time building through automated checks. In experiments, a compliant agent failed a held‑back test while a rule‑breaking agent passed, showing that a passing test suite can be gamed. The study also demonstrates that the automated check mechanism, rather than interface‑locking alone, is crucial for reliable rebuilds, and that multi‑level verification catches errors that single‑level checks miss.
arXiv:2603.01209v3 Announce Type: replace Abstract: In CodeAct, language-model agents write Python that calls tools and use execution feedback to choose actions. Persistent runtimes preserve Python v...
arXiv:2607. 28545v2 Announce Type: replace-cross Abstract: Large language models can write, patch, and search code, but oncall root cause analysis (RCA) demands something different: reasoning over noisy metrics, logs, traces, and source code, starting from ambiguous user-facing reports, often hours after the incident began.
Software engineering (SWE) agents resolve repository-level issues through long trajectories that grow increasingly expensive as context accumulates. Failed runs tend to be longer and exhibit redundant exploration or looping, suggesting that some failures may be detectable before completion.
InfraDev is a new benchmark comprising 36 tasks derived from real-world pull requests to Megatron‑LM, vLLM, and verl, designed to test coding agents on infrastructure development for training, inference, and reinforcement learning across many GPUs. Each task requires agents to reimplement multi‑file changes and run correctly, with grading that includes hidden probes and a prompt exposing only the interfaces used. The benchmark reveals that current frontier models resolve only about 39% of tasks, with probes rejecting a third of submissions that pass official tests, and that structural and state errors dominate failures.