arXiv:2608. 16742v1 Announce Type: cross Abstract: Large Language Models (LLMs) have achieved remarkable progress in code generation, yet ensuring correctness in complex, repository-level tasks remains challenging.
Large Language Models (LLMs) have achieved remarkable progress in code generation, yet ensuring correctness in complex, repository-level tasks remains challenging. Existing approaches often use generated tests as static post-hoc validators, which limits their ability to guide implementation and may introduce misleading feedback when the tests themselves are incomplete or incorrect.
arXiv:2509. 24148v3 Announce Type: replace-cross Abstract: Test-Driven Development (TDD) is a widely adopted practice that requires developers to create and execute tests alongside implementation.
E2E-SWE is a benchmark that tests large language models’ ability to create complete, functional software repositories from scratch. It includes 186 tasks across 11 programming languages, each requiring an agent to build an installable project based solely on a natural‑language specification and an empty workspace, while passing a hidden test suite. The benchmark was crafted by software engineers and LLMs, then refined through iterative verification by autonomous agents to ensure clarity and solvability.
arXiv:2607. 02057v1 Announce Type: cross Abstract: In recent years, it has become increasingly evident that large language models (LLMs) and autonomous agents raise the level of abstraction in software development by shifting the focus from writing precise procedures to expressing intents and goals.
The paper introduces RobustTests, a framework that improves reinforcement learning for code generation by synthesizing test cases from faulty code and refining rewards with a dense, stepwise function. It uses validator agents and behavioral clustering to filter out invalid or redundant tests, and incorporates pass‑rate‑based rewards to counter hallucination noise. Experiments on CodeContests and LiveCodeBench show that fine‑tuning Qwen3‑32B with RobustTests yields a 3% absolute performance gain over baseline methods.
arXiv:2607. 08124v1 Announce Type: cross Abstract: The behavior of an LLM agent is determined not only by the underlying model, but also by its harness: the executable program that constructs context, invokes tools, verifies intermediate results, and recovers from failures.
The paper presents a systematic analysis of five state‑of‑the‑art automated program repair agents, tracing their decision‑making across 500 real‑world repair tasks. It finds that while the agents perform well on simple fixes, they struggle with logic‑intensive bugs, often producing verbose, overfitted patches that pass tests without addressing root causes. Key bottlenecks identified include poor test generation, limited regression test selection, and reliance on primitive tooling without access to debuggers or advanced program analysis tools.
Schr"odinger's Repository (Schr"odingerRepo) is an evaluation framework that tests coding agents on dynamically instantiated repository representations to mitigate data leakage from static repository benchmarks. It transforms test repositories through four levels—problem statement reconstruction, namespace remapping, intra-file layout reordering, and functionality-preserving code rewriting—to obscure familiar cues while preserving executable behavior. Experiments on popular LLMs using SWE-bench Verified and SWE-QA show that removing these cues consistently degrades performance and increases interaction costs, mainly due to harder repository exploration and localization.
arXiv:2506. 13932v3 Announce Type: replace-cross Abstract: The rise of large language models (LLMs) has led to dramatic improvements across a wide range of natural language tasks.
arXiv:2607. 08983v1 Announce Type: cross Abstract: While autonomous coding agents have significantly advanced automated test generation, they remain fundamentally limited by lazy generation, a phenomenon where agents prematurely terminate tasks and systematically avoid complex programmatic logic, resulting in inadequate code coverage.
CovR is an agentic framework that automates testbench generation for hardware verification by combining self-reflection loops with simulation-based feedback to maximize coverage. It builds a large dataset of 16,514 specification–RTL reasoning tuples and uses reinforcement learning with tool-derived rewards to train a student model, achieving high coverage scores on VerilogEval, RTLLM V2.0, and CVDP. When deployed as a plug-in stimulus engine, CovR boosts coverage by nearly 19% and improves mutation detection while uncovering previously undetected failures.
arXiv:2607. 02469v1 Announce Type: cross Abstract: Software tests and code evolve together: a code change should be followed by new or updated tests that record the new software behavior.