arXiv Machine Learning
RealSWE: A Compositional Evaluation of Coding Agents under Realistic User Requests
Read the original on arXiv Machine Learning →The Flow has not summarised this story yet — read it at arXiv Machine Learning.
Related stories
arXiv Machine Learning
Jun 30
SWE-INTERACT: Reimagining SWE Benchmarks as User-Driven Long-Horizon Coding Sessions
arXiv:2606. 30573v1 Announce Type: new Abstract: We introduce SWE-Interact, a new testbed for evaluating coding agents on multi-turn, interactive, user-driven software engineering tasks.
arXiv Machine Learning
Oct 8
Coding-Agent Benchmarks Should Match Their Users' Task Flows
arXiv:2610.09633v1 Announce Type: new Abstract: The evaluation of coding agents generally strives to be as realistic as possible. In our study, we collect 4,782 agent sessions of real software engine...
Hugging Face Trending Papers
Oct 7
Coding-Agent Benchmarks Should Match Their Users' Task Flows
The evaluation of coding agents generally strives to be as realistic as possible. In our study, we collect 4,782 agent sessions of real software engineers in JetBrains IDEs, which we call Production S...
arXiv Computation and Language
Sep 30
LoLBench: Evaluating Coding Agents with Long-Horizon Proposals on Large Software Systems
arXiv:2609.37143v1 Announce Type: cross Abstract: Modern coding agents can deliver increasingly large repository-level changes, and recent benchmarks reflect this by emphasizing long-horizon tasks wi...
arXiv AI
Oct 12
Don't Guess, Just Ask: Evaluating Interactive Requirement Clarification for Repository-Level Software Development
The paper introduces ReqClarifyBench, an interactive benchmark for evaluating how well large language models (LLMs) can clarify vague user intent into clear requirements in repository-level software development. It includes 341 tasks across game, miniapp, and web development, each with a vague intent, a human-validated requirement document, and code evaluation artifacts, and uses LLM-based user simulators with diverse personas. The study evaluates six LLMs on clarification effectiveness and requirement quality, finding that stronger clarification does not always yield better documents, performance varies by persona, and coding-agent harnesses do not improve document quality over base LLMs.
arXiv Computation and Language
Sep 15
CALICO: A Human-Centered, Codebook-Aligned System for Annotation
arXiv:2609.14726v1 Announce Type: cross Abstract: Large language models are increasingly used to scale codebook-based annotation in scientific research, but existing workflows provide limited support...