arXiv AI
Constraint Decay: The Fragility of LLM Agents in Backend Code Generation
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv AI
Oct 28
Benchmarking LLM Agents for Behaviorally Adequate Test Generation from Software Requirements
The paper introduces Req2Test-Bench, a benchmark of 5,086 instances linking natural‑language Gherkin requirements to developer‑written test implementations across five languages and four BDD frameworks. It evaluates eight agent–model–reasoning setups, finding that while the best configuration (Codex with DeepSeek‑V4.1‑Flash) produces passing tests for 82.8% of cases, only 33.8% are behaviorally adequate, as measured by kill‑completeness. The study highlights diminishing returns from scaling agents, the impact of under‑specified requirements, and the potential of requirement‑grounded harnesses to improve test generation.
Hugging Face Trending Papers
Oct 27
Benchmarking LLM Agents for Behaviorally Adequate Test Generation from Software Requirements
As large language model agents increasingly compile natural-language requirements into repository-level code, requirement-to-test generation grows in importance to provide validation constraints (or h...
arXiv AI
Oct 1
E2E-SWE: Benchmarking LLMs on Building Working Codebases from Scratch
E2E-SWE is a benchmark that tests large language models’ ability to create complete, functional software repositories from scratch. It includes 186 tasks across 11 programming languages, each requiring an agent to build an installable project based solely on a natural‑language specification and an empty workspace, while passing a hidden test suite. The benchmark was crafted by software engineers and LLMs, then refined through iterative verification by autonomous agents to ensure clarity and solvability.
arXiv AI
Jul 20
Evaluating LLM-Based 0-to-1 Software Generation in End-to-End CLI Tool Scenarios
arXiv:2604. 06742v2 Announce Type: replace-cross Abstract: The evolution of Large Language Models (LLMs) has catalyzed a paradigm shift towards intent-driven software development, where autonomous agents are expected to design and deliver complete, runnable software systems from scratch.
arXiv AI
Oct 2
Code That Works, Environments That Don't: Measuring Environment Reproducibility in AI-Generated Software
The paper investigates how well large language models specify the software environments needed to run AI-generated code. Using a new agent protocol and a three‑layer dependency framework, the authors evaluate three coding agents across four languages and fifty tasks, finding that dependency specifications are often inconsistent, redundant, or incomplete. Agreement on dependency sets is as low as 7% for identical tasks, and newer agents show no improvement, indicating that environment specification remains a significant, unaddressed challenge in code generation.
arXiv AI
Oct 15
EvoWeb: A Benchmark for Coding Agents on Evolving Web Applications
arXiv:2610.16402v1 Announce Type: new Abstract: Web application development often proceeds through successive requests that extend an existing implementation. Yet web-generation benchmarks primarily...