REAP: Automatic Curation of Coding Agent Benchmarks from Interactive Production Usage
arXiv:2604. 01527v4 Announce Type: replace-cross Abstract: Production deployment of AI coding agents requires fast, reproducible evaluation signals.
arXiv:2608. 06640v1 Announce Type: cross Abstract: The widespread integration of AI coding assistants offers undeniable boosts to engineering velocity.
arXiv:2604. 01527v4 Announce Type: replace-cross Abstract: Production deployment of AI coding agents requires fast, reproducible evaluation signals.
The paper examines how AI coding agents are evolving beyond simple autocomplete to perform complex tasks such as repository inspection, multi-file editing, tool execution, test writing, pull request creation, and long-duration work with minimal supervision. It highlights that while these agents boost coding activity, significant bottlenecks remain in review, integration, testing, security, deployment, and production operations, and that the economics of software development are shifting toward variable token, tool, sandbox, CI, and rework costs. The authors synthesize recent research and industry data to propose four engineering concepts—Agentic SDLC Throughput Paradox, Production-Qualified Change, Verification Tax, and an Agentic SDLC Control Plane—to guide the allocation of autonomy within cost, reliability, and human-attention constraints, ultimately reframing the research focus to production-qualified value per dollar, reviewer-hour, and operational risk.
arXiv:2606. 08676v1 Announce Type: cross Abstract: AI coding assistants have significantly improved developer productivity by automatically suggesting code that aligns with user intent, and many of these tools are now integrated directly into Integrated Development Environments (IDEs).
arXiv:2609.12708v2 Announce Type: replace-cross Abstract: AI coding assistants are becoming co-authors of production software, yet their evaluation centers on functional correctness, leaving open whe...
arXiv:2607. 03691v2 Announce Type: replace-cross Abstract: Coding agents, autonomous systems that use large language models (LLMs) to resolve software engineering tasks, rely on agent harness: a middleware layer in between a developer and a large language model that orchestrates system prompts, tool execution, context management, and iterative reasoning loops.
arXiv:2610.00425v1 Announce Type: cross Abstract: Code generation has emerged as a central capability of large language models, with coding agents now able to produce functionally correct software pr...
arXiv:2604. 06742v2 Announce Type: replace-cross Abstract: The evolution of Large Language Models (LLMs) has catalyzed a paradigm shift towards intent-driven software development, where autonomous agents are expected to design and deliver complete, runnable software systems from scratch.
arXiv:2605. 17548v2 Announce Type: replace-cross Abstract: Code review has evolved for decades, from informal peer checking to today's pull request (PR) workflows, yet it remains a largely manual and cognitively demanding process.
arXiv:2607. 25130v1 Announce Type: cross Abstract: Imperfections in AI-generated code require that software developers modify the generated code manually, or by re-prompting an AI programming assistant.
arXiv:2608.22638v1 Announce Type: cross Abstract: Copying a function from a chat window into an editor takes less than a second. For many uses of AI coding tools, that speed is the point; in settings...
arXiv:2604. 05150v2 Announce Type: replace-cross Abstract: We study compiled AI, a paradigm in which large language models generate executable code artifacts during a compilation phase, after which workflows execute deterministically without further model invocation.
arXiv:2607. 03691v1 Announce Type: cross Abstract: Coding agents, autonomous systems that use large language models (LLMs) to resolve software engineering tasks, rely on agentic scaffolding: a middleware layer in between a developer and a large language model that orchestrates system prompts, tool execution, context management, and iterative reasoning loops.