MotorSense is the first large‑scale dataset that pairs egocentric video with surface electromyography (EMG) and inertial measurement unit (IMU) data, covering 100 hours of unscripted bimanual manipulation. The dataset enables cross‑modal pre‑training that aligns visual and motor signals, improving downstream tasks such as action recognition, 3D hand reconstruction, and 3D contact reconstruction. Additionally, EMG signals can condition a video diffusion model to generate fine‑grained hand‑manipulation videos.
arXiv:2610.33685v1 Announce Type: cross Abstract: Vision-Language-Action (VLA) models are bottlenecked by the scarcity of robot data. Egocentric human manipulation videos offer an abundant, diverse,...
The paper introduces GALA, a Geometry-Aware Latent Action modeling framework that enhances image-based latent actions with 3D end‑effector motion. It proposes the Unified End‑effector Motion Representation (UEMR) to preserve fine‑grained motion while improving cross‑embodiment generalizability. Experiments show GALA effectively models generalizable fine‑grained motions across embodiments, achieving high success rates in RoboCasa-GR1 and real‑world tasks.
Robotic manipulation with dexterous hands is a cornerstone of Embodied AI, yet its progress is stifled by the high cost of collecting embodiment-aware teleoperation data. While abundant egocentric videos of human hands offer a scalable alternative, the profound discrepancies in appearance, articulation, and camera viewpoints between human and robotic data raise significant challenges for co-training.
The paper introduces P³-WAM, a unified world action model for egocentric dexterous manipulation that integrates 3D perception, action-conditioned prediction, and goal-directed planning. It operates in three modes—Perceive, Predict, and Plan—using a mixture-of-transformers architecture that shares base weights and adapts to modality-specific inputs. Experiments demonstrate that unified training improves performance across all three capabilities, outperforming specialized baselines on 3D hand reconstruction and action-conditioned video prediction while producing more instruction‑adherent manipulation videos.
arXiv:2610.06970v1 Announce Type: new Abstract: Egocentric bimanual hand pose estimation is important for virtual interaction, wearable control, and rehabilitation, but visual observations are often...
arXiv:2606. 00174v1 Announce Type: cross Abstract: Electromyography (EMG) directly reflects muscle activation and is a key sensing modality for gesture recognition, prosthetic control, and wearable interaction.
EgoJEPA is a three‑stage framework that learns embodiment‑agnostic representations from egocentric human videos, eliminating the need for hand‑crafted retargeting mappings. It first pre‑trains latent actions via visual prediction conditioned on hand‑motion differences, then models future visual dynamics and latent actions jointly, and finally adapts to specific robots with lightweight action adapters. Experiments on five manipulation tasks across three robot embodiments show a 74% average success rate versus 65% for a retargeting baseline, while cutting cross‑embodiment wall‑clock cost by 47.2%.
TouchéWorld is a framework that merges gesture-driven interaction with camera-controlled video world models, enabling users to explore generated environments using both camera navigation and continuous hand motion. Starting from an initial image and a scene-description prompt, the system generates video conditioned on temporally aligned camera and hand controls, built from joint camera–hand training data derived from heterogeneous egocentric videos. The approach incorporates spatial hand-skeleton conditioning, numerical hand-motion features, causal adaptation, and few-step distillation to support chunk-level online generation, while TouchéBench evaluates hand-motion following, camera–hand coordination, online interaction, and object response.
arXiv:2608. 14028v1 Announce Type: cross Abstract: Dexterous manipulation is a fundamental capability for embodied intelligence, but scaling it remains difficult because robot demonstrations are expensive to collect and action spaces vary across embodiments.
arXiv:2610.10288v1 Announce Type: new Abstract: Large-scale egocentric human interaction data is becoming an important source of physical supervision for embodied learning, yet video alone leaves the...
arXiv:2610.27946v1 Announce Type: new Abstract: Egocentric action transfer aims to decouple the semantic action from egocentric observations and reproduce them in new visual contexts, enabling scalab...
MyoSem is an EMG–action semantic alignment framework that maps hand electromyography signals and multi‑view natural‑language action descriptions into a shared feature space for bidirectional retrieval. It constructs semantic prototypes from text, encodes EMG with local activation‑aware representations, and aggregates evidence using semantic queries. Evaluated on multiple datasets and protocols, MyoSem outperforms existing methods in bidirectional EMG‑text retrieval, demonstrating its effectiveness in moving EMG hand action understanding toward language‑queryable semantic retrieval.