arXiv:2507. 22080v2 Announce Type: replace-cross Abstract: Acquiring high-quality instruction-code pairs is essential for training Large Language Models for code generation.
CCTU is a new benchmark designed to evaluate large language models (LLMs) on their ability to use tools under complex constraints. It includes 200 test cases that average seven constraint types and 4,700‑token prompts, covering resource, behavior, toolset, and response dimensions. An executable validation module performs step‑level checks, and nine state‑of‑the‑art LLMs were tested, revealing that none exceed a 20% task completion rate when strict constraints are enforced, with frequent violations and limited self‑refinement.
The paper introduces Multi-step SWE, a benchmark consisting of 69 software engineering tasks that require a coding agent to perform 5–15 sequential, multi-hour engineering requests on a single codebase. Each step builds on the code produced in the previous step, making early design decisions and errors propagate throughout the chain. The benchmark covers four engineering work families, nine programming languages, and includes both from-scratch builds and existing open-source projects, with each step verified against a source of truth. Across nine state‑of‑the‑art models, the best achieves only an 18.5% solve rate, and 38 tasks remain unsolved by any model.
arXiv:2606. 30573v1 Announce Type: new Abstract: We introduce SWE-Interact, a new testbed for evaluating coding agents on multi-turn, interactive, user-driven software engineering tasks.
arXiv:2601.19747v2 Announce Type: replace-cross Abstract: In the rapidly evolving field of Electronic Design Automation (EDA), the deployment of Large Language Models (LLMs) for Register-Transfer Lev...
The paper introduces Many-Tier Instruction Hierarchy (ManyIH), a new framework for resolving conflicts among instructions with arbitrarily many privilege levels in large language model agents. It presents ManyIH-Bench, a benchmark featuring 853 agentic tasks that require navigating up to 12 levels of conflicting instructions across 46 real-world agents. Experiments show current models achieve only about 40% accuracy when instruction conflict scales, highlighting a gap in fine-grained, scalable conflict resolution.
UserSWE is a new benchmark that evaluates coding agents’ ability to fix repository-level bugs through multi‑turn conversations with dynamic users, providing execution‑grounded verdicts after each turn. The benchmark transforms repository histories and existing single‑turn datasets into multi‑turn tasks, using a persona chain derived from 20,859 development sessions to simulate user interactions. Eleven agents were tested on 72 tasks from 42 repositories, revealing that agent rankings shift with the interaction budget and that single‑turn performance alone does not capture interactive coding capability.
SWE-Journey is a new benchmark designed to evaluate coding assistants in more realistic settings, focusing on long-horizon tasks and multi-turn interactions. It introduces a weak-to-strong synthesis pipeline to create extended coding challenges and a user-simulation agent based on four real user personas to mimic authentic code-assistance dialogues. The benchmark reveals that while models succeed on over 75% of tests with software architects, they manage less than 25% with non-coders, highlighting a significant gap in current assistant capabilities.
SWE-DIVE is a dynamic, interactive framework for evaluating coding agents in multi-turn scenarios. It introduces a user simulator that generates varied behavioral patterns and a conversation validator that checks whether the simulator’s outputs meet initial requirements, guiding the agent toward task completion. The framework runs within a Harbor environment that orchestrates the agent, simulator, and validator to assess success on coding tasks.
arXiv:2605.06445v2 Announce Type: replace-cross Abstract: Large Language Model (LLM) agents demonstrate strong performance in autonomous code generation under loose specifications. However, productio...
E2E-SWE is a benchmark that tests large language models’ ability to create complete, functional software repositories from scratch. It includes 186 tasks across 11 programming languages, each requiring an agent to build an installable project based solely on a natural‑language specification and an empty workspace, while passing a hidden test suite. The benchmark was crafted by software engineers and LLMs, then refined through iterative verification by autonomous agents to ensure clarity and solvability.
arXiv:2607. 13705v1 Announce Type: new Abstract: As Large Language Models (LLMs) evolve into autonomous agents, the need for unified evaluation infrastructure becomes critical.