arXiv AI
UserSWE: Evaluating Coding Agents in Multi-Turn Conversations with Dynamic Users
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
Hugging Face Trending Papers
Oct 7
Coding-Agent Benchmarks Should Match Their Users' Task Flows
The evaluation of coding agents generally strives to be as realistic as possible. In our study, we collect 4,782 agent sessions of real software engineers in JetBrains IDEs, which we call Production S...
arXiv Machine Learning
Jun 30
SWE-INTERACT: Reimagining SWE Benchmarks as User-Driven Long-Horizon Coding Sessions
arXiv:2606. 30573v1 Announce Type: new Abstract: We introduce SWE-Interact, a new testbed for evaluating coding agents on multi-turn, interactive, user-driven software engineering tasks.
arXiv Machine Learning
Oct 8
Coding-Agent Benchmarks Should Match Their Users' Task Flows
arXiv:2610.09633v1 Announce Type: new Abstract: The evaluation of coding agents generally strives to be as realistic as possible. In our study, we collect 4,782 agent sessions of real software engine...
arXiv Computation and Language
Oct 12
SWE-Duel: Towards a Self-Scaling Arena for Coding Agents
SWE‑Duel proposes a self‑scaling benchmark for coding agents, where tasks and oracles are generated at evaluation time to prevent data leakage and adapt difficulty to participant performance. In the arena, two agents take turns as Red and Blue: Red proposes a feature, injects a subtle bug, and must repair it blind, while Blue reviews the pull request without knowing the bug and must pass all tests. A tournament of 11 model‑harness participants on 15 repositories across five languages showed GPT‑5.5/Codex CLI as the top solver and Claude Opus 4.8/mini‑swe‑agent as the top task author, with newer model versions producing harder tasks for older ones.
Hugging Face Trending Papers
Oct 8
SWE-Duel: Towards a Self-Scaling Arena for Coding Agents
SWE‑Duel proposes a self‑scaling benchmark for coding agents, where tasks and oracles are generated at evaluation time to prevent data leakage and adapt difficulty to the agents’ capabilities. In the arena, two agents take turns modifying a real repository: the Red agent adds a feature, introduces a subtle bug, and the Blue agent must fix it without knowing the bug exists. A tournament of 11 model‑harness participants on 15 repositories in five languages showed GPT‑5.5 as the best solver and Claude Opus 4.8 as the best task author, with newer model versions producing harder tasks for older ones.
arXiv AI
1d ago
SWE-DIVE: Multi-Turn Coding Agent Evaluation with Dynamic User Simulation
arXiv:2610.30047v1 Announce Type: cross Abstract: Coding agents are typically evaluated using one-shot, well-formulated prompts. Benchmarks such as SWE-bench use existing GitHub issues as input for a...