arXiv:2609.35841v1 Announce Type: cross Abstract: Mutation testing evaluates test-suite adequacy by injecting synthetic faults into program code. However, traditional rule-based tools often generate...
arXiv:2509. 24148v3 Announce Type: replace-cross Abstract: Test-Driven Development (TDD) is a widely adopted practice that requires developers to create and execute tests alongside implementation.
arXiv:2607. 22880v1 Announce Type: cross Abstract: Recent advances in large language models (LLMs) have driven growing interest in using LLMs to automate test generation.
arXiv:2604. 06742v2 Announce Type: replace-cross Abstract: The evolution of Large Language Models (LLMs) has catalyzed a paradigm shift towards intent-driven software development, where autonomous agents are expected to design and deliver complete, runnable software systems from scratch.
arXiv:2609.10397v1 Announce Type: cross Abstract: Exception Related Code (ERC), which includes throw statements, conditions (if statements) that guard those throw statements, and try/catch blocks, is...
arXiv:2607. 02469v1 Announce Type: cross Abstract: Software tests and code evolve together: a code change should be followed by new or updated tests that record the new software behavior.
The paper introduces SWE-Flux, a repository‑level benchmark designed to test large language models’ ability to reason about runtime behavior. It contains 480 execution‑grounded instances from 12 real Python repositories, with gold answers automatically harvested from instrumented test executions. Evaluation of five LLMs shows the task remains difficult, with the best model achieving only 37% accuracy, and the benchmark can generate challenging variants through input perturbation.
The paper introduces Req2Test-Bench, a benchmark of 5,086 instances linking natural‑language Gherkin requirements to developer‑written test implementations across five languages and four BDD frameworks. It evaluates eight agent–model–reasoning setups, finding that while the best configuration (Codex with DeepSeek‑V4.1‑Flash) produces passing tests for 82.8% of cases, only 33.8% are behaviorally adequate, as measured by kill‑completeness. The study highlights diminishing returns from scaling agents, the impact of under‑specified requirements, and the potential of requirement‑grounded harnesses to improve test generation.
arXiv:2507. 22580v2 Announce Type: replace-cross Abstract: Automated Program Repair (APR) seeks to automatically correct software bugs without requiring human intervention.
E2E-SWE is a benchmark that tests large language models’ ability to create complete, functional software repositories from scratch. It includes 186 tasks across 11 programming languages, each requiring an agent to build an installable project based solely on a natural‑language specification and an empty workspace, while passing a hidden test suite. The benchmark was crafted by software engineers and LLMs, then refined through iterative verification by autonomous agents to ensure clarity and solvability.
SWE‑Nexus is a fully automated pipeline that builds repository‑level implementation tasks by analyzing test execution. It uses four construction agents to select tests, trace production function bodies, and generate behavioral specifications, retaining scopes only when all expected scoring points pass and masking the traced bodies triggers failures. The benchmark covers 600 tasks from 105 repositories in six languages, and the best configuration resolves 37.8% of tasks in a single attempt, demonstrating the system’s ability to construct tasks from validated behavior beyond individual code changes.
arXiv:2610.09079v1 Announce Type: cross Abstract: Large language models equipped with development environments have moved code generation toward repository-scale construction, yet building complete r...