arXiv Machine Learning

SWE-Tester: Training Open-Source LLMs for Issue Reproduction in Real-World Repositories

arXiv AI

Jul 14

SWE-MERA: A Dynamic Benchmark for Agenticly Evaluating Large Language Models on Software Engineering Tasks

arXiv:2507. 11059v3 Announce Type: replace-cross Abstract: The rapid advancement of Large Language Models (LLMs) in software engineering has revealed critical limitations in existing benchmarks, particularly the widely used SWE-bench dataset.

By Pavel Adamenko, Mikhail Ivanov, Aidar Valeev, Rodion Levichev, Pavel Zadorozhny, Ivan Lopatin, Dmitry Babaev, Alena Fenogenova, Valentin Malykh

llms agents benchmarks

arXiv Machine Learning

Jun 9

RepoLaunch: Automating Build and Management of Code Repositories across Languages and Platforms

arXiv:2603. 05026v2 Announce Type: replace-cross Abstract: Language model (LM) agents have driven substantial progress in automated software engineering (SWE), yet building and testing software repositories at scale remains a largely manual and labor-intensive bottleneck.

By Kenan Li, Rongzhi Li, Linghao Zhang, Qirui Jin, Liao Zhu, Xiaosong Huang, Geng Zhang, Yikai Zhang, Shilin He, Chengxing Xie, Xin Zhang, Zijian Jin, Bowen Li, Chaoyun Zhang, Yu Kang, Yufan Huang, Elsie Nallipogu, Saravan Rajmohan, Qingwei Lin, Dongmei Zhang

llms agents benchmarks

arXiv AI

Jul 14

TENET: One Step Toward Test-Driven Development for Repository-Level Code Generation

arXiv:2509. 24148v3 Announce Type: replace-cross Abstract: Test-Driven Development (TDD) is a widely adopted practice that requires developers to create and execute tests alongside implementation.

By Yiran Hu, Nan Jiang, Shanchao Liang, Yi Wu, Lin Tan

llms agents

arXiv AI

Jul 2

SWE-Doctor: Guiding Software Engineering Agents with Runtime Diagnosis from Multi-Faceted Bug Reproduction Tests

arXiv:2607. 00990v1 Announce Type: cross Abstract: Large language model (LLM)-based software engineering agents are increasingly developed to resolve software issues by generating patches from issue reports and code repositories.

By Yaoqi Guo, Yang Liu, Jie M. Zhang, Yun Ma, Yiling Lou, Zhenpeng Chen

llms agents benchmarks

arXiv Machine Learning

Jul 30

MindForge: Teaching Small Language Models Whole-Life-Cycle Software Engineering via Source-Free Program Synthesis

arXiv:2607. 27146v1 Announce Type: cross Abstract: Coding agents have made substantial progress on software engineering tasks that modify existing codebases, including bug fixing and feature implementation.

By Yihao Chen, Shi Chang, Khaled Chawa, Feng Lin, Boyuan Chen, Shaowei Wang, Ahmed E. Hassan

llms agents fine-tuning benchmarks

arXiv AI

Jul 2

LLVM-Bench: Benchmarking and Advancing Large Language Models for LLVM Compiler Issue Resolution

arXiv:2607. 00700v1 Announce Type: cross Abstract: LLVM is a widely used compiler infrastructure whose scale and complexity make issue resolution labor-intensive and challenging.

By Zhao Tian, Yingquan Zhao, Chenyao Suo, Meng Wang, Junjie Chen

llms agents benchmarks

arXiv AI

Jun 3

Toward Training Superintelligent Software Agents through Self-Play SWE-RL

arXiv:2512. 18552v3 Announce Type: replace-cross Abstract: While current software agents powered by large language models (LLMs) and agentic reinforcement learning (RL) can boost programmer productivity, their training data (e.

By Yuxiang Wei, Zhiqing Sun, Emily McMilin, Jonas Gehring, David Zhang, Gabriel Synnaeve, Daniel Fried, Lingming Zhang, Sida Wang

llms agents reinforcement-learning benchmarks

arXiv AI

Jul 3

Prompt Coverage Adequacy

arXiv:2607. 02057v1 Announce Type: cross Abstract: In recent years, it has become increasingly evident that large language models (LLMs) and autonomous agents raise the level of abstraction in software development by shifting the focus from writing precise procedures to expressing intents and goals.

By Florian Tambon, Michael Konstantinou, Cedric Richter, Charles Chenouard, Mark Harman, Mike Papadakis

llms agents

arXiv AI

Jul 9

AnyPoC: Universal Proof-of-Concept Test Generation for Scalable LLM-Based Bug Detection

arXiv:2604. 11950v2 Announce Type: replace-cross Abstract: While recent LLM-based agents can identify many candidate bugs in source code, their reports remain static hypotheses that require manual validation, limiting the practicality of automated bug detection.

By Zijie Zhao, Chenyuan Yang, Weidong Wang, Yihan Yang, Ziqi Zhang, Lingming Zhang

llms rag agents benchmarks safety

arXiv AI

Jul 28

Do Coverage and Mutation Scores of LLM-Generated Test Suites Correlate with Their Effectiveness? (Replicability Study)

arXiv:2607. 22880v1 Announce Type: cross Abstract: Recent advances in large language models (LLMs) have driven growing interest in using LLMs to automate test generation.

By Junda Zhao, Shurui Zhou, Eldan Cohen

llms

arXiv AI

Aug 5

PAIChecker: Uncovering and Checking PR-Issue Misalignment in SWE-Bench-Like Benchmarks

arXiv:2607. 28587v2 Announce Type: replace-cross Abstract: SWE-bench-like benchmarks are widely used for evaluating LLM's issue resolution capability.

By Manyi Wang, Junjielong Xu, Pinjia He

llms agents benchmarks

arXiv AI

Sep 2

WiseSpec: Requirements-Driven Agents for Code Generation

WiseSpec is a requirements‑driven agent framework designed to improve repository‑level code generation. It automatically builds structured, information‑rich requirements, evaluates their quality via execution‑based tests, and iteratively refines them to better guide code generation. Experiments show WiseSpec outperforms all baselines, achieving an average 13.17% improvement in %Resolved.

By Zhao Tian

llms agents

Related stories