FeatureLoop: Benchmarking Coding Agents under Evolving Requirements

FeatureLoop is a benchmark that evaluates coding agents on 61 human‑reviewed tasks drawn from 34 repositories in five programming languages. Each task tracks a development goal through iterative feedback—adding or clarifying requirements and reporting failures—while cumulative executable checks monitor new behavior, repairs, and preservation of earlier functionality. In tests of 12 state‑of‑the‑art models, the best agent completed 72.1% of tasks with both feedback types, but only 15.4% of failures were recovered within two revisions, highlighting issues such as incomplete repairs, regressions, and token inefficiency.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

• agents
• benchmarks