TEX is a new test‑time scaling framework that uses execution‑driven cross‑validation to generate and refine tests for language‑model agents. By running agents in parallel and cross‑executing each test against every patch, TEX builds a $K imes K$ feedback matrix that guides objective refinement without sharing verbose traces. On the SWT‑Bench Verified benchmark, TEX reaches 86.4% accuracy—setting a new state‑of‑the‑art and improving over single‑agent baselines by 9.0%—and even random selections from its outputs outperform the previous best score.
arXiv:2602.18998v2 Announce Type: replace Abstract: LLM agents are increasingly expected to operate as general-purpose systems that resolve real-world user requests, yet their dynamic scaling behavio...
TestJack is a scalable framework that audits large language model (LLM) coding agents by generating new tests for each trial, focusing on prompt requirements that may be violated. It keeps only the tests passed by the ground‑truth patch and re‑examines failures, providing replayable evidence for each confirmed error. In experiments across six model backends and five benchmarks, TestJack found that 34.4% of trials deemed correct actually violated task requirements, reducing the overall resolution rate from 50.6% to 33.2%.
arXiv:2509. 24148v3 Announce Type: replace-cross Abstract: Test-Driven Development (TDD) is a widely adopted practice that requires developers to create and execute tests alongside implementation.
SWEPatchTest is a benchmark that evaluates software tests by measuring their ability to reproduce issues, accept correct patches, and reject incorrect ones across 619 tasks and 82,579 candidate patches. It introduces the Joint Validation Rate metric, revealing that even the best configurations only achieve 9.21% success when all three criteria are met. Using these validation scores to select demonstrations for supervised fine‑tuning can roughly triple the Joint Validation Rate and enhance downstream repair performance on multiple SWE‑bench datasets.
The paper introduces R4P, a reasoning‑based supervision method for software agents that eliminates the need for test execution by using a group‑wise training objective to verify multiple patches simultaneously. R4P achieves 72.2% accuracy on the SWE‑bench patch verification task, matching proprietary models, and enables the creation of an execution‑free scaffold called Mini‑SE. Mini‑SE, trained purely with reinforcement learning via R4P, improves Pass@1 from 26.2% to 32.8% over the baseline Qwen3‑32B, demonstrating R4P’s practical utility and scalable performance.
arXiv:2608.22191v1 Announce Type: new Abstract: Software-engineering agents solve repository-level tasks through long, stochastic tool-use trajectories, and repeated attempts often find fixes missed...
arXiv:2608. 16742v1 Announce Type: cross Abstract: Large Language Models (LLMs) have achieved remarkable progress in code generation, yet ensuring correctness in complex, repository-level tasks remains challenging.
The paper introduces Req2Test-Bench, a benchmark of 5,086 instances linking natural‑language Gherkin requirements to developer‑written test implementations across five languages and four BDD frameworks. It evaluates eight agent–model–reasoning setups, finding that while the best configuration (Codex with DeepSeek‑V4.1‑Flash) produces passing tests for 82.8% of cases, only 33.8% are behaviorally adequate, as measured by kill‑completeness. The study highlights diminishing returns from scaling agents, the impact of under‑specified requirements, and the potential of requirement‑grounded harnesses to improve test generation.
WatchPoint is a simulated‑user system that generates and runs diagnostic scripts against a live web application, producing structured observations to guide coding model retries. Unlike prior methods that rely on screenshots or non‑executable metrics, WatchPoint operates on Web‑Bench—a benchmark of 50 multi‑file web projects with 1,000 sequential tasks verified by deterministic end‑to‑end tests. It recovers 57.6% of diagnosed tasks, matching a human tester’s 54.5% recovery rate, and identifies when such simulated feedback is beneficial or should be withheld.
Large Language Models (LLMs) have achieved remarkable progress in code generation, yet ensuring correctness in complex, repository-level tasks remains challenging. Existing approaches often use generated tests as static post-hoc validators, which limits their ability to guide implementation and may introduce misleading feedback when the tests themselves are incomplete or incorrect.
As large language model agents increasingly compile natural-language requirements into repository-level code, requirement-to-test generation grows in importance to provide validation constraints (or h...
SWE‑Nexus is a fully automated pipeline that builds repository‑level implementation tasks by analyzing test execution. It uses four construction agents to select tests, trace production function bodies, and generate behavioral specifications, retaining scopes only when all expected scoring points pass and masking the traced bodies triggers failures. The benchmark covers 600 tasks from 105 repositories in six languages, and the best configuration resolves 37.8% of tasks in a single attempt, demonstrating the system’s ability to construct tasks from validated behavior beyond individual code changes.