arXiv:2604. 07341v2 Announce Type: replace-cross Abstract: Most repository-level code translation and validation techniques have been evaluated on a single source-target programming language (PL) pair, owing to the complex engineering effort required to adapt new PL pairs.
E2E-SWE is a benchmark that tests large language models’ ability to create complete, functional software repositories from scratch. It includes 186 tasks across 11 programming languages, each requiring an agent to build an installable project based solely on a natural‑language specification and an empty workspace, while passing a hidden test suite. The benchmark was crafted by software engineers and LLMs, then refined through iterative verification by autonomous agents to ensure clarity and solvability.
arXiv:2608. 09072v1 Announce Type: cross Abstract: Large language model-powered coding agents are increasingly used to modify existing code repositories, for example, by adding features or fixing bugs.
arXiv:2610.09079v1 Announce Type: cross Abstract: Large language models equipped with development environments have moved code generation toward repository-scale construction, yet building complete r...
The paper introduces SWE Refactor Bench, a benchmark of 20 whole‑repository migrations aimed at testing whether coding agents can autonomously perform long‑horizon stack migrations. It employs a three‑stage evaluation—Migration Audit, Behavioural Tests, and Agentic Verification—to assess both migration completeness and behavioural correctness, revealing that only 5.4 % of runs succeed across all stages. The study shows that agents struggle with migration tasks, especially language rewrites, and highlights the need for more robust coding‑agent capabilities.
arXiv:2609.38269v1 Announce Type: cross Abstract: Coding agents are increasingly asked to build software rather than patch it, yet benchmarks for from-scratch repository construction are mostly limit...
arXiv:2609.37143v1 Announce Type: cross Abstract: Modern coding agents can deliver increasingly large repository-level changes, and recent benchmarks reflect this by emphasizing long-horizon tasks wi...
The paper introduces ADFD‑Migrate, a method that extracts a latent declarative representation of code—an annotated data‑flow diagram (ADFD)—to aid large‑scale repository migration. By using an LLM to infer the source ADFD from repository context and guiding target‑language generation with dependency‑aware chunking, the approach improves porting soundness and completeness. Evaluated on 50 Fortran repositories, the system achieves high behavioral agreement and a superior migration outcome index compared to baseline translation methods.
arXiv:2607. 28591v1 Announce Type: cross Abstract: Scaling coding agents requires a continuing supply of executable data for training, benchmarking, and continuous evaluation.
arXiv:2609.22068v1 Announce Type: new Abstract: Training capable coding agents via reinforcement learning (RL) requires diverse tasks with reliable verifiers. Open-source codebases offer a rich sourc...
FeatureArc is a new benchmark that evaluates coding agents on the evolution of software features, requiring agents to extend a feature across multiple stages while preserving earlier functionality. The benchmark uses a pipeline that constructs tasks from real repository histories, validates fail‑to‑pass and pass‑to‑pass tests, and covers four programming languages. In tests on 73 tasks, agents performed well on the first stage (62.7% success) but dropped to 28.5% when the entire feature was evaluated, with inherited defects causing most failures.
Modern coding agents can deliver increasingly large repository-level changes, and recent benchmarks reflect this by emphasizing long-horizon tasks with large reference implementations. Many benchmarks...