arXiv Computer Vision
Oct 22

Multimodal Embodied Agents: A Unified Capability-centric Perspective from Computer-Use to Robot-Use

The paper introduces Multimodal Embodied Agents (MMEAs), goal‑directed systems that combine multimodal task reasoning with physical action and adapt decisions based on feedback. It proposes the PAPAV framework, which defines five core capabilities—Perceive, Anticipate, Plan, Act, Verify—common to computer‑use agents, robotic systems, and MMEAs. The authors analyze 64 benchmarks, noting that most evaluations focus on Act while Anticipate and Verify are rarely assessed, highlighting gaps and open challenges in designing reliable physical agents.

By Yanzhe Chen, Ziyi Yang, Jifeng Zhu, Qiming Huang, Ruihe An, Peiyao Xu, Hesen Yang, Runda Liu, Chang Gong, Zhijun Cao, Zechen Bai, Wenzheng Zeng, Yiqi Lin, Guoqiang Liang, Kevin Yuchen Ma, Kevin Qinghong Lin, Mike Zheng Shou