DeepSWE: Measuring Frontier Coding Agents on Original, Long-Horizon Engineering Tasks
arXiv:2607. 07946v1 Announce Type: cross Abstract: DeepSWE is a benchmark of 113 original, long-horizon software engineering tasks for evaluating coding agents.
ExpertCrowd‑SWE (EC‑SWE) is a benchmark comprising 100 expert‑authored, multi‑file software engineering tasks designed to evaluate autonomous coding agents on large, unfamiliar codebases. Each task is a real change request from a practising engineer, paired with a sealed reference solution, a task‑specific grader that awards partial credit, and must‑pass blockers that mimic a maintainer’s merge criteria. The benchmark covers 87 open‑source projects across 11 languages, and the final 100 tasks were selected for their difficulty against current frontier models, with seven models evaluated over 3,495 runs.
arXiv:2607. 07946v1 Announce Type: cross Abstract: DeepSWE is a benchmark of 113 original, long-horizon software engineering tasks for evaluating coding agents.
FrontierCS 2 is a new benchmark comprising 48 open‑ended computer‑science research tasks that place language‑model agents inside a research environment, allowing them to submit incremental patches to a working repository. Each patch is scored by a judge on a development suite, while the final evaluation uses a held‑out suite and a private reference derived from the original authors’ code. The benchmark covers 13 domains, includes a private set of over 100 future tasks, and evaluates 12 agents, with Claude Opus 5.5 achieving the highest pass rate of 58%.
arXiv:2607. 06411v1 Announce Type: cross Abstract: Developers increasingly delegate real maintenance work to product-grade coding agents, and many state tasks in their native language, in the style of a customer request rather than a curated English issue.
arXiv:2609.38269v1 Announce Type: cross Abstract: Coding agents are increasingly asked to build software rather than patch it, yet benchmarks for from-scratch repository construction are mostly limit...
SWE‑Duel proposes a self‑scaling benchmark for coding agents, where tasks and oracles are generated at evaluation time to prevent data leakage and adapt difficulty to participant performance. In the arena, two agents take turns as Red and Blue: Red proposes a feature, injects a subtle bug, and must repair it blind, while Blue reviews the pull request without knowing the bug and must pass all tests. A tournament of 11 model‑harness participants on 15 repositories across five languages showed GPT‑5.5/Codex CLI as the top solver and Claude Opus 4.8/mini‑swe‑agent as the top task author, with newer model versions producing harder tasks for older ones.
arXiv:2610.09079v1 Announce Type: cross Abstract: Large language models equipped with development environments have moved code generation toward repository-scale construction, yet building complete r...
arXiv:2610.17757v1 Announce Type: new Abstract: Large language model agents can follow complex human instructions to solve real-world tasks. In practice, however, initial instructions are often incom...
The paper introduces agent‑native primitives—reusable building blocks that embed a language model within a resource to apply it contextually—and HEELO, a harness‑engineering framework that composes these primitives into task‑specific graphs, verifies outcomes, and iteratively refines or adds primitives. Using eight primitive families, the authors demonstrate significant performance gains across multiple benchmarks, raising Terminal‑Bench 4.0 resolution from 37.3% to 55.5% and improving pass rates on ToolBench and GAIA‑Text‑103. When integrated with HEELO, Claude Opus 5.5 and GPT‑6.1 Sol achieve higher completion counts, faster runtimes, and lower costs on 46 real‑world tasks.
Developers increasingly delegate real maintenance work to product-grade coding agents, and many state tasks in their native language, in the style of a customer request rather than a curated English issue. Existing repository-level agentic benchmarks do not measure this setting: their task statements are English by design.
arXiv:2609.11987v1 Announce Type: cross Abstract: An agentic coding system couples a language model to a harness: the tools, prompts and control flow that turn a chat model into an autonomous softwar...
SWE-Skills-Bench is a benchmark that evaluates the impact of individual agent skills on real-world software engineering tasks. It uses 98 public skills paired with specific GitHub repositories, generating 3,920 paired evaluations across four agent-model configurations. The study finds that most skills are ineffective, with only a few providing modest gains while some are harmful, raising the overall pass ratio from 76.6% to 76.9%.
arXiv:2610.32441v1 Announce Type: new Abstract: Large language model (LLM) agents are increasingly capable of supporting scientific workflows, but their capabilities across diverse research activitie...