The paper introduces ORACLE‑SWE, a study that systematically evaluates how different intermediate information signals—such as Reproduction Test, Regression Test, Edit Location, Execution Context, and API Usage—contribute to the success of language‑model agents on software engineering tasks. It quantifies the impact of both oracle‑provided signals and those extracted by agents, and measures the performance gap between current agent‑found signals and their oracle counterparts. The authors release code and a dataset to support future signal‑analysis research in this area.
arXiv:2606. 18168v1 Announce Type: cross Abstract: Software practitioners increasingly use AI coding agents that generate test code alongside production code in open source pull requests (PRs).
SWE-Critix is a post‑training pipeline that creates generalist verifiers for coding‑agent trajectories. It uses supervised fine‑tuning followed by reinforcement learning to train 30B models on CodeForge‑Preview data, enabling the verifier to predict completion labels and generate natural‑language explanations. The approach raises classification accuracy from 57% to 78% on a held‑out test set.
arXiv:2607. 03691v2 Announce Type: replace-cross Abstract: Coding agents, autonomous systems that use large language models (LLMs) to resolve software engineering tasks, rely on agent harness: a middleware layer in between a developer and a large language model that orchestrates system prompts, tool execution, context management, and iterative reasoning loops.
arXiv:2507. 11059v3 Announce Type: replace-cross Abstract: The rapid advancement of Large Language Models (LLMs) in software engineering has revealed critical limitations in existing benchmarks, particularly the widely used SWE-bench dataset.
arXiv:2607. 04786v1 Announce Type: cross Abstract: Test maintenance is a critical, yet costly, activity - particularly as codebases rapidly evolve.
Chronos is a test‑time framework that transforms historical pull requests into structured experience cards linked by a typed graph of code, developer intent, and organizational relations. It uses semantic search and weighted multi‑hop expansion to retrieve relevant cards, guiding both patch generation and selection by LLM‑based code agents. The approach improves resolution rates across multiple LLM backbones on SWE‑Bench and related benchmarks, and human evaluation shows a higher number of useful cards retrieved compared to flat semantic retrieval.
E2E-SWE is a benchmark that tests large language models’ ability to create complete, functional software repositories from scratch. It includes 186 tasks across 11 programming languages, each requiring an agent to build an installable project based solely on a natural‑language specification and an empty workspace, while passing a hidden test suite. The benchmark was crafted by software engineers and LLMs, then refined through iterative verification by autonomous agents to ensure clarity and solvability.
arXiv:2609.08149v1 Announce Type: new Abstract: SWE-Bench Pro has emerged as a standard benchmark for evaluating software engineering agents on challenging repository-level tasks. However, our analys...
arXiv:2606. 08500v1 Announce Type: cross Abstract: Software engineering agents (SWE agents) increasingly work through tool-mediated trajectories in real repositories, yet their behavior remains difficult to characterize in concrete, observable terms.
arXiv:2509. 24148v3 Announce Type: replace-cross Abstract: Test-Driven Development (TDD) is a widely adopted practice that requires developers to create and execute tests alongside implementation.
arXiv:2607. 28591v1 Announce Type: cross Abstract: Scaling coding agents requires a continuing supply of executable data for training, benchmarking, and continuous evaluation.
arXiv:2606. 30573v1 Announce Type: new Abstract: We introduce SWE-Interact, a new testbed for evaluating coding agents on multi-turn, interactive, user-driven software engineering tasks.