← Back to all news
arXiv Computer Vision By Zeyi Zhang, Zixi Kang, Ruijie Zhao, Yusen Feng, Biao Jiang, Hanyu Ji, Libin Liu

ProAct: Harnessing Streaming Motion Generation and Agentic Reasoning for Real-Time Embodied Social Interaction

Read the original on arXiv Computer Vision →

ProAct is a dual‑system framework for real‑time embodied social interaction that separates a low‑latency Behavioral System, which streams multimodal interaction and generates continuous non‑verbal motion, from a slower Cognitive System that performs long‑horizon social reasoning and produces proactive intentions. The Cognitive System uses an efficient memory mechanism and a user‑motivation prediction module to decide when to intervene, while the Behavioral System translates these intentions into fluid motion via an intention‑conditioned streaming flow‑matching generator with a disentangled ControlNet branch. The framework is deployed on a physical humanoid robot and validated through real‑world user studies, motion‑generation benchmarks, and a new ProActBench benchmark for proactive trigger detection and restraint.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computer Vision.

Related stories

arXiv AI
Oct 15

Proactive Physical AI through Continuous Observation and Timely Intervention

arXiv:2610.16967v1 Announce Type: cross Abstract: Physical AI is commonly formulated around an explicitly specified task: given a goal or instruction, a system perceives, reasons, and acts. In many r...

By Zhenhua Yang, Jiafeng Liang, Xingyu Zhou, Yi Jiang, Xiayu Cao, Zui Chen, Luqi Wang, Haiyang Sun, Yufei Xiang, Chang Li, Rongchuan Mu, Bowen Chi, Zihe Tong, Mingyu Li, Huijie Ma, Yiqiao Yang, Baoqi Ren, Zhiming Ma, Zhihao Zhu, Jiahao Ji, Zhuoyao Wang, Qianyu Wang, Xiangyu Liu, Wang Zixiang, Shunliang Zhang, Ming Liu, Xiaocheng Feng, Duyu Tang, Bing Qin
roboticsdata-engineering
More like this →
arXiv Computer Vision
Sep 22

Cognitive Action Reasoning for Proactive Robots from Human-Centered Multimodal Observations

The paper introduces ProAction, a multimodal dataset of 10,000 samples comprising visual, audio, and text inputs across 12 daily-life scenarios, designed to support the Proactive Robot Action Reasoning (ProRobo) problem. It presents a two-stage human-in-the-loop annotation pipeline that incorporates appraisal and Theory-of-Mind considerations to generate cognitively grounded high-level action labels. The authors benchmark multimodal large language models and propose MMC2Act, showing that training on ProAction significantly improves proactive action reasoning compared to general-purpose models.

By Zhihao Gu, Kechao Zhu, Yuanfeng Wu, Mohan Liu, Ankit Kumar Shaw, ChenDong Hong, Xuanyu Chen, Dengchen Mei, Xu Tianyi, Lin Wang
llmsroboticsmultimodalbenchmarks
More like this →
arXiv Computer Vision
Aug 27

StreamPI: Streaming Multimodal Temporal Modeling for Vision-Language-Action Models

StreamPI introduces a streaming multimodal temporal modeling framework that enhances Vision‑Language‑Action models by adding temporal reasoning without extra parameters. It anchors each visual observation and language instruction pair as a temporal unit, using bidirectional attention for cross‑modal fusion and causal attention for autoregressive streaming inference. The method employs random‑interval streaming training to improve robustness and leverages the LLM backbone’s length extrapolation to inherit pretrained weights, achieving superior performance over pi0.5 on real‑robot and simulation tasks.

By Zhe Liu, Jinghua Hou, Yuxiang Lu, Zhenya Yang, Xianzhe Fan, Junwei Luo, Junyi Li, Ruihua Han, Zhi Hou, Hengshuang Zhao
llmsroboticsmultimodalbenchmarks
More like this →