arXiv Machine Learning
SWE-Tester: Training Open-Source LLMs for Issue Reproduction in Real-World Repositories
Read the original on arXiv Machine Learning →The Flow has not summarised this story yet — read it at arXiv Machine Learning.
Related stories
arXiv AI
Jul 14
SWE-MERA: A Dynamic Benchmark for Agenticly Evaluating Large Language Models on Software Engineering Tasks
arXiv:2507. 11059v3 Announce Type: replace-cross Abstract: The rapid advancement of Large Language Models (LLMs) in software engineering has revealed critical limitations in existing benchmarks, particularly the widely used SWE-bench dataset.
arXiv Machine Learning
Jun 9
RepoLaunch: Automating Build and Management of Code Repositories across Languages and Platforms
arXiv:2603. 05026v2 Announce Type: replace-cross Abstract: Language model (LM) agents have driven substantial progress in automated software engineering (SWE), yet building and testing software repositories at scale remains a largely manual and labor-intensive bottleneck.
arXiv AI
Jul 14
TENET: One Step Toward Test-Driven Development for Repository-Level Code Generation
arXiv:2509. 24148v3 Announce Type: replace-cross Abstract: Test-Driven Development (TDD) is a widely adopted practice that requires developers to create and execute tests alongside implementation.
arXiv AI
Jul 2
SWE-Doctor: Guiding Software Engineering Agents with Runtime Diagnosis from Multi-Faceted Bug Reproduction Tests
arXiv:2607. 00990v1 Announce Type: cross Abstract: Large language model (LLM)-based software engineering agents are increasingly developed to resolve software issues by generating patches from issue reports and code repositories.
arXiv AI
Oct 1
E2E-SWE: Benchmarking LLMs on Building Working Codebases from Scratch
E2E-SWE is a benchmark that tests large language models’ ability to create complete, functional software repositories from scratch. It includes 186 tasks across 11 programming languages, each requiring an agent to build an installable project based solely on a natural‑language specification and an empty workspace, while passing a hidden test suite. The benchmark was crafted by software engineers and LLMs, then refined through iterative verification by autonomous agents to ensure clarity and solvability.
arXiv AI
Sep 24
Can LLMs Reason About Runtime Behavior? A Repository-Level Dynamic Benchmark
The paper introduces SWE-Flux, a repository‑level benchmark designed to test large language models’ ability to reason about runtime behavior. It contains 480 execution‑grounded instances from 12 real Python repositories, with gold answers automatically harvested from instrumented test executions. Evaluation of five LLMs shows the task remains difficult, with the best model achieving only 37% accuracy, and the benchmark can generate challenging variants through input perturbation.