ROSTrans: Empirically Benchmarking LLMs for ROS1-to-ROS2 Migration

ROTrans is a benchmark that evaluates large language models (LLMs) on the task of migrating code from ROS1 to ROS2. It contains 108 scaffolded completion tasks derived from over 70 open‑source ROS1 repositories, organized into a hierarchical taxonomy that tests interface‑level, behavioral‑level, and system‑level correctness. Each task comes with a verified ROS2 reference implementation, a runtime test suite that exercises topics, services, actions, and parameters, and a container for reproducibility, allowing candidates to be scored by inserting only their target files into a fixed reference package. "whyItMatters":"The study reveals that current frontier LLMs achieve less than 10.9% pass@1 on these tasks, highlighting significant gaps in LLM performance for complex robotics software development and underscoring the need for execution‑guided repair techniques to improve migration success."

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

• llms
• robotics
• benchmarks