BuildBench introduces a realistic benchmark for evaluating large language model agents on the task of compiling open‑source software (OSS). It includes diverse OSS projects that lack clear build instructions, have undocumented dependencies, and may require source patching or script modification. The authors also present OSS‑BUILD‑AGENT, a baseline LLM‑based agent that retrieves build instructions effectively and achieves state‑of‑the‑art performance on the benchmark.
arXiv:2610.07832v1 Announce Type: cross Abstract: Large language models (LLMs) equipped with terminal access have demonstrated strong capabilities in automating software engineering tasks. However, e...
arXiv:2607. 27146v1 Announce Type: cross Abstract: Coding agents have made substantial progress on software engineering tasks that modify existing codebases, including bug fixing and feature implementation.
arXiv:2603. 20075v2 Announce Type: replace-cross Abstract: Compilers are critical to modern computing, yet fixing compiler bugs is difficult.
E2E-SWE is a benchmark that tests large language models’ ability to create complete, functional software repositories from scratch. It includes 186 tasks across 11 programming languages, each requiring an agent to build an installable project based solely on a natural‑language specification and an empty workspace, while passing a hidden test suite. The benchmark was crafted by software engineers and LLMs, then refined through iterative verification by autonomous agents to ensure clarity and solvability.
ProgramGym introduces a scalable framework for training long-horizon program reproduction agents, comprising Program-Env and Program-Agent components. Program-Env builds verifiable training environments from repositories via a feedback loop that repairs sandbox builds and improves black-box tests, yielding 1,648 environments across C, C++, Go, and Rust. Program-Agent employs a role-isolated workflow of analysis, implementation, and review with revision management, boosting performance on ProgramBench and raising Qwen3.8-27B’s accuracy from 34.87% to 42.84% and surpassing flagship models with fine-tuning.
arXiv:2607. 00700v1 Announce Type: cross Abstract: LLVM is a widely used compiler infrastructure whose scale and complexity make issue resolution labor-intensive and challenging.
arXiv:2604. 06742v2 Announce Type: replace-cross Abstract: The evolution of Large Language Models (LLMs) has catalyzed a paradigm shift towards intent-driven software development, where autonomous agents are expected to design and deliver complete, runnable software systems from scratch.
arXiv:2610.09079v1 Announce Type: cross Abstract: Large language models equipped with development environments have moved code generation toward repository-scale construction, yet building complete r...
arXiv:2609.37143v1 Announce Type: cross Abstract: Modern coding agents can deliver increasingly large repository-level changes, and recent benchmarks reflect this by emphasizing long-horizon tasks wi...
arXiv:2607. 02370v1 Announce Type: cross Abstract: Compiler missed optimizations refer to cases in which compilers failed to optimize certain code.
The paper introduces Multi-step SWE, a benchmark consisting of 69 software engineering tasks that require a coding agent to perform 5–15 sequential, multi-hour engineering requests on a single codebase. Each step builds on the code produced in the previous step, making early design decisions and errors propagate throughout the chain. The benchmark covers four engineering work families, nine programming languages, and includes both from-scratch builds and existing open-source projects, with each step verified against a source of truth. Across nine state‑of‑the‑art models, the best achieves only an 18.5% solve rate, and 38 tasks remain unsolved by any model.
arXiv:2507. 11059v3 Announce Type: replace-cross Abstract: The rapid advancement of Large Language Models (LLMs) in software engineering has revealed critical limitations in existing benchmarks, particularly the widely used SWE-bench dataset.