arXiv AI
6d ago

Gym-Anything for Robotics: Scaling physical simulations with digital agents

Gym-Anything for Robotics is a framework that uses coding agents to automatically create realistic robotics simulations, addressing the scalability bottleneck of expert-driven simulation design. By selecting tasks based on U.S. GDP data and employing an adversarial creation-audit loop, the authors built RoboWorld, a benchmark of 25 simulations with 25 test and 175 training RL environments. The paper also introduces new techniques—such as in-context learning, privileged teachers, self-assisted learning, and code policies—to improve frontier models, all of which significantly boost performance on the benchmark.

By Pranjal Aggarwal, Sean Welleck
arXiv AI
Jun 30

StarDojo: Benchmarking Open-Ended Behaviors of Agentic Multimodal LLMs in Production-Living Simulations with Stardew Valley

arXiv:2507. 07445v3 Announce Type: replace Abstract: Autonomous agents navigating human society must master both production activities and social interactions, yet existing benchmarks rarely evaluate these skills simultaneously.

By Weihao Tan, Changjiu Jiang, Yu Duan, Mingcong Lei, Jiageng Li, Yitian Hong, Xinrun Wang, Bo An
arXiv AI
Sep 7

RoboSPA: Can VLA Models Go Beyond Simple Scenes and Short-Horizon Tasks?

RoboSPA is a large-scale robotic manipulation dataset and benchmark designed to evaluate Vision‑Language‑Action models on fine‑grained spatial reasoning and long‑horizon procedural planning. It contains 10 task categories, 56 base tasks, and 280 variants across five difficulty levels, with 527K trajectories collected from multiple embodiments and scenes. The benchmark introduces diagnostic metrics beyond binary success, revealing that current VLA models struggle with complex spatial relations, precise execution, and memory‑intensive planning.

By Zhenxuan Fan, Bo Zhang, Yutong Lin, Yuqian Yuan, Juekai Lin, Liang Liang, Zhuoyi Huang, Wenqiao Zhang, Juncheng Li, Siliang Tang, Jun Xiao, Yueting Zhuang