Agent4RE is a self‑refining multi‑agent framework designed for end‑to‑end software requirements engineering (RE). It coordinates specialized agents and includes two iterative improvement loops, and introduces a new real‑world dataset, RE‑E2E, for evaluating RE workflows from elicitation to generation. The framework, along with two enhanced variants that use autonomous self‑refinement or structured human feedback, consistently outperforms baseline prompting methods across eight large language models, achieving higher text‑based metrics and superior human and LLM‑as‑a‑judge ratings.
By Yongjian Tang, Linhan Li, Thomas Runkler
LLM-based agents excel at software engineering tasks where an existing codebase provides context, but constructing a program from scratch remains fundamentally harder. Recent benchmarks such as ProgramBench quantify this gap: given only natural-language documentation and an execute-only binary as a behavioral oracle, even frontier models solve fewer than 1% of instances.
The paper investigates how well large language models specify the software environments needed to run AI-generated code. Using a new agent protocol and a three‑layer dependency framework, the authors evaluate three coding agents across four languages and fifty tasks, finding that dependency specifications are often inconsistent, redundant, or incomplete. Agreement on dependency sets is as low as 7% for identical tasks, and newer agents show no improvement, indicating that environment specification remains a significant, unaddressed challenge in code generation.
By Bhanu Prakash Vangala, Tanu Malik
E2E-SWE is a benchmark that tests large language models’ ability to create complete, functional software repositories from scratch. It includes 186 tasks across 11 programming languages, each requiring an agent to build an installable project based solely on a natural‑language specification and an empty workspace, while passing a hidden test suite. The benchmark was crafted by software engineers and LLMs, then refined through iterative verification by autonomous agents to ensure clarity and solvability.
By Hantian Ding, Chloe Bi, Jiacheng Zhu, John Yang, Matt Deitke, Pengcheng Yin, Zijian Wang, Rui Hou
The paper introduces ReqClarifyBench, an interactive benchmark for evaluating how well large language models (LLMs) can clarify vague user intent into clear requirements in repository-level software development. It includes 341 tasks across game, miniapp, and web development, each with a vague intent, a human-validated requirement document, and code evaluation artifacts, and uses LLM-based user simulators with diverse personas. The study evaluates six LLMs on clarification effectiveness and requirement quality, finding that stronger clarification does not always yield better documents, performance varies by persona, and coding-agent harnesses do not improve document quality over base LLMs.
By Mingyu Chen, Zhiqing Zhong, Manyi Wang, Linxi Liang, Rouyu Chen, Pinjia He
arXiv:2512. 22256v2 Announce Type: replace-cross Abstract: Software issue resolution aims to address real-world issues in software repositories based on natural language descriptions provided by users, and represents a key aspect of software maintenance.
By Zhonghao Jiang, David Lo, Zhongxin Liu
arXiv:2608. 09072v1 Announce Type: cross Abstract: Large language model-powered coding agents are increasingly used to modify existing code repositories, for example, by adding features or fixing bugs.
By Xin Zhou, Chun Yong Chong, Kisub Kim, Yun Peng, Rui Shu, Zihan Wu, Xu Han, Guowen Yuan, Zeyang Zhuang, Jounghoon Kim, Jeongjin Ju, Seongmin Ju, Taein Yoon, David Lo
arXiv:2610.35333v1 Announce Type: cross
Abstract: Code agents have become increasingly capable of performing software engineering tasks through interaction with software environments. However, agents...
By Huanyu Liu, Yezheng Cheng, Jinliang You, Tianxiao Huang, Linyu Li, Zheng Fang, Peng Cao, Ge Li
arXiv:2607. 02057v1 Announce Type: cross Abstract: In recent years, it has become increasingly evident that large language models (LLMs) and autonomous agents raise the level of abstraction in software development by shifting the focus from writing precise procedures to expressing intents and goals.
By Florian Tambon, Michael Konstantinou, Cedric Richter, Charles Chenouard, Mark Harman, Mike Papadakis
arXiv:2509. 24148v3 Announce Type: replace-cross Abstract: Test-Driven Development (TDD) is a widely adopted practice that requires developers to create and execute tests alongside implementation.
By Yiran Hu, Nan Jiang, Shanchao Liang, Yi Wu, Lin Tan
arXiv:2501. 07892v2 Announce Type: replace-cross Abstract: Large language models (LLMs) have shown strong performance in automated code generation, with few-shot prompting widely used for its simplicity and effectiveness.
By Shengsheng Zhou, Shuai Wang, Liang Ding, Yibing Zhan, Yong Luo, Zheng He, Fu Lin, Dapeng Tao
Large Language Models (LLMs) have achieved remarkable progress in code generation, yet ensuring correctness in complex, repository-level tasks remains challenging. Existing approaches often use generated tests as static post-hoc validators, which limits their ability to guide implementation and may introduce misleading feedback when the tests themselves are incomplete or incorrect.