arXiv AI

ORDER: A Fictitious-World Benchmark for Domain-Adaptive Embodied AI

The paper introduces ORDER, a fictitious-world benchmark designed to evaluate domain-adaptive embodied AI. ORDER consists of a synthetic 342,069-token corpus defining a self-consistent physics, a 500-question knowledge test (ORDER‑BENCH), and a compositional spatial task (ORDER‑SPATIAL) that requires ordering objects for safe manipulation. The benchmark demonstrates that models like GPT‑4.1 perform poorly without adaptation, while small models improve significantly after continual pre‑training, and that performance on ORDER‑SPATIAL better predicts real plan quality than knowledge-test accuracy.

arXiv AI

6d ago

Do World Models Make Better Robots? A Survey of Evaluation Benchmarks for Predictive Embodied Intelligence

The paper surveys 160 benchmarks from 2017‑2026 that evaluate predictive embodied intelligence, categorising them into policy suites, embodied agents, world‑model evaluation, and prediction‑to‑action bridges. It finds that most benchmarks are model‑agnostic, rarely compare Vision‑Language‑Action policies to world models, and seldom turn predictions into executed actions. The authors argue that the lack of benchmarks designed to directly test the closed‑loop advantage of world models prevents the field from answering whether such models truly improve robotic performance.

By Gaytri Jena, Kapil Wanaskar, Vinija Jain, Aman Chadha, Vasu Sharma, Amitava Das

agents robotics multimodal benchmarks

arXiv AI

Aug 13

G0.5: One Autoregressive Stream for Robot Reasoning and Action

arXiv:2608. 11739v1 Announce Type: cross Abstract: The prevailing recipe for Vision-Language-Action (VLA) models couples a pretrained VLM with a separately trained flow-matching action expert.

By Yicheng Liu, Zibin Dong, Baijun Ye, Tianyuan Yuan, Tao Jiang, Anqi Yang, Shicheng Cao, Haonan Liu, Yue Sun, Zihan Guo, Xiao Liu, Dong Ke, Changxun Pan, Chenru Wu, Tailai Cheng, Xiaoshu Ren, Xinlei Zhang, Jianning Cui, Zijie Zhao, Haoyu Zhang, Kaiming Xu, Haodong Yang, Bowen Zhang, Jiahui Niu, Shaoting Zhu, Shiduo Zhang, Hang Zhao

llms nlp robotics fine-tuning multimodal benchmarks

arXiv AI

Jul 7

Cortex: A Bidirectionally Aligned Embodied Agent Framework for Long-horizon Manipulation

arXiv:2607. 05377v1 Announce Type: cross Abstract: While recent Vision-Language-Action (VLA) models show promise toward generalist manipulation policies, they struggle with long-horizon tasks due to their Markovian nature-relying solely on current observations.

By Jiaqi Peng, Xiqian Yu, Delin Feng, Yuqiang Yang, Wenzhe Cai, Jing Xiong, Ganlin Yang, Jinliang Zheng, Jiafei Cao, Xueyuan Wei, Jiangmiao Pang, Yuan Shen, Tai Wang

agents robotics fine-tuning multimodal

arXiv AI

Jun 30

Hierarchical Experimentalist Agents

arXiv:2606. 29315v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly used to take actions in the real world and support human decision-making, yet most agents rely on parametric knowledge, fixed post-training data, retrieval, or search.

By Abhranil Chandra, Sankaran Vaidyanathan, Utsav Dhanuka, Varun Gandhi, Scott Niekum

llms agents benchmarks

arXiv Computer Vision

Sep 22

An Unexpected Robot Policy: Early Evaluations of GPT-6 Astra on RoboDojo and Beyond

arXiv:2609.24170v1 Announce Type: new Abstract: Embodied AI systems are often organized into System 1 and System 2. System 1 is typically a pretrained policy that generates actions at high frequency,...

By Wenbo Zhang, Kaixuan Wang, Yutao Ouyang, Xiaoyu Huang, Liyang Li, Kailun Su, Weiyang Jin, Wenhao Chai, Haotian Liang, Zhiyang Dou, Yue Chen, Tianxing Chen

llms robotics fine-tuning

arXiv Computation and Language

1d ago

Video2Skill: From Streaming Experience to Reusable Embodied Skills

arXiv:2609.36691v1 Announce Type: new Abstract: Manipulation behaviors vary widely across objects and scenes, but they share a small set of reusable skills, and planning with these skills helps embod...

By Jianshu Zhang, Ce Zhang, Xiyuan Yang, Chenwei Xu, Haoran Lu, Yijiang Li, Yaqi Xie, Katia P. Sycara, Han Liu

llms agents robotics fine-tuning multimodal benchmarks

arXiv AI

Sep 10

Learning Counterfactual World Models for Embodied Reasoning under Partial Observability

arXiv:2609.05834v1 Announce Type: new Abstract: World models promise a general route to embodied intelligence: learn predictive dynamics once, then reason, plan, and act with them. Increasingly, the...

By Todd Y. Zhou, Daniel Zhang

robotics multimodal

arXiv AI

1d ago

Spatial Memory Agent: Experience-Grounded Procedural Memory for Spatial Intelligence

arXiv:2608.12743v3 Announce Type: replace Abstract: Spatial intelligence is becoming a foundation for embodied agents, robotic planning, and multimodal assistants. To improve the spatial reasoning ab...

By Haokai Zhang, Yuhang Ding, Yunshu Zhou, Xinze Du, Shengtao Zhang, Zhiyue Zhao, Yuling Xi, Hao Chen

agents reinforcement-learning robotics fine-tuning multimodal benchmarks

arXiv AI

Jun 11

Embodied-R1.5: Evolving Physical Intelligence via Embodied Foundation Models

arXiv:2606. 11324v1 Announce Type: cross Abstract: We introduce Embodied-R1.

By Yifu Yuan, Yaoting Huang, Xianze Yao, Yutong Li, Shuoheng Zhang, Linqi Han, Pengyi Li, Jiangeng Sun, Wenting Jia, Zhao Zhang, Yuhao Liu, Ruihao Liao, Yucheng Hu, Qiyu Wu, Yuxiao Li, Zibin Dong, Fei Ni, Yan Zheng, Shuyang Gu, Yi Ma, Hongyao Tang, Han Hu, Jianye Hao

llms reinforcement-learning robotics fine-tuning multimodal benchmarks

arXiv AI

Aug 10

Capek 0.5: An Execution-Centric Vision-Language Model for Embodied Intelligence

arXiv:2608. 06756v1 Announce Type: new Abstract: Vision-language models are increasingly serving as the reasoning core of embodied agents.

By Ying Chen, Weizhen Li, Zhe Hu, Zhenjiang Li, Rui Jiang, Zhifeng Gu, Lihuang Fang, Jiangping Liu, Lei Yi, Jie Chen

llms agents reinforcement-learning robotics efficiency multimodal benchmarks

arXiv AI

Jul 7

Do Vision-Language-Action Models Mean What They Say? On the Role of Faithfulness in Embodied Reasoning

arXiv:2607. 04681v1 Announce Type: cross Abstract: Embodied Chain-of-Thought has emerged as a promising mechanism to enhance robot decision-making and interpretability in black-box Vision-Language Action (VLA) models.

By Matthew Foutter, Matteo Cercola, Lena Wild, Yunshan Wang, Michelle Li, Daniele Gammelli, Marco Pavone

agents reinforcement-learning robotics multimodal benchmarks safety

arXiv AI

Aug 7

SkillMemo: Expert-guided Skill Memory Framework for Compositional Embodied Manipulation

arXiv:2608. 05970v1 Announce Type: cross Abstract: Embodied visuomotor models, including Diffusion Policy (DP) and Vision-Language-Action (VLA) models, have demonstrated promising performance on robotic manipulation benchmarks.

By Changyuan Wang, Chubin Zhang, Zhenyu Wu, Runhao Li, Angyuan Ma, Ke Chao, Yinan Liang, Xiuwei Xu, Ziwei Wang, Yansong Tang, Jiwen Lu

diffusion computer-vision robotics multimodal benchmarks

Related stories