ROTrans is a benchmark that evaluates large language models (LLMs) on the task of migrating code from ROS1 to ROS2. It contains 108 scaffolded completion tasks derived from over 70 open‑source ROS1 repositories, organized into a hierarchical taxonomy that tests interface‑level, behavioral‑level, and system‑level correctness. Each task comes with a verified ROS2 reference implementation, a runtime test suite that exercises topics, services, actions, and parameters, and a container for reproducibility, allowing candidates to be scored by inserting only their target files into a fixed reference package. "whyItMatters":"The study reveals that current frontier LLMs achieve less than 10.9% pass@1 on these tasks, highlighting significant gaps in LLM performance for complex robotics software development and underscoring the need for execution‑guided repair techniques to improve migration success."
arXiv:2610.09079v1 Announce Type: cross Abstract: Large language models equipped with development environments have moved code generation toward repository-scale construction, yet building complete r...
arXiv:2604. 06742v2 Announce Type: replace-cross Abstract: The evolution of Large Language Models (LLMs) has catalyzed a paradigm shift towards intent-driven software development, where autonomous agents are expected to design and deliver complete, runnable software systems from scratch.
E2E-SWE is a benchmark that tests large language models’ ability to create complete, functional software repositories from scratch. It includes 186 tasks across 11 programming languages, each requiring an agent to build an installable project based solely on a natural‑language specification and an empty workspace, while passing a hidden test suite. The benchmark was crafted by software engineers and LLMs, then refined through iterative verification by autonomous agents to ensure clarity and solvability.
arXiv:2602. 09464v2 Announce Type: replace-cross Abstract: Vericoding refers to the generation of formally verified code from rigorous specifications.
The paper introduces SWE Refactor Bench, a benchmark of 20 whole‑repository migrations aimed at testing whether coding agents can autonomously perform long‑horizon stack migrations. It employs a three‑stage evaluation—Migration Audit, Behavioural Tests, and Agentic Verification—to assess both migration completeness and behavioural correctness, revealing that only 5.4 % of runs succeed across all stages. The study shows that agents struggle with migration tasks, especially language rewrites, and highlights the need for more robust coding‑agent capabilities.
arXiv:2610.34633v1 Announce Type: cross Abstract: Repository-level code migration rewrites an entire project in a new programming language while preserving its observable behavior. It is challenging...
arXiv:2608. 14863v1 Announce Type: cross Abstract: LLM-based coding agents have advanced rapidly on single-process SWE tasks, with frontier models now clustering in the high-70s on SWE-bench Verified.
Large language models are deployed as long-horizon agents that compose diverse tools across MCP services into executable workflows. Existing MCP environment-scaling methods are data-centric: they firs...
AI agents are increasingly used for programming, but do not provide any guarantee on the correctness of generated code. Verified code generation, in which an agent produces both an implementation and a machine-checked proof of its specification, offers a stronger path toward trustworthy AI-generated software.
The paper introduces ADFD‑Migrate, a method that extracts a latent declarative representation of code—an annotated data‑flow diagram (ADFD)—to aid large‑scale repository migration. By using an LLM to infer the source ADFD from repository context and guiding target‑language generation with dependency‑aware chunking, the approach improves porting soundness and completeness. Evaluated on 50 Fortran repositories, the system achieves high behavioral agreement and a superior migration outcome index compared to baseline translation methods.
arXiv:2609.18959v1 Announce Type: new Abstract: LLM code-generation systems usually choose a target programming language before decoding and treat that choice as fixed. We show that, for language-fle...
arXiv:2605.29054v3 Announce Type: replace-cross Abstract: Migrating a machine-learning codebase across frameworks--for example from PyTorch to JAX--is a routine but costly engineering task that is fu...