World models offer a data-driven alternative to traditional simulators for robotics, with applications spanning policy evaluation, improvement, and planning. All of these uses depend on faithful 3D ge...
XGenAct is a world action model that encodes RGB observations, robot actions, metric depth, surface normals, and functional role segmentation as RGB videos using deterministic codecs. It trains a single video diffusion transformer with a unified objective, sampling different perception and action tasks to learn temporal prediction across multiple spatial modalities without separate heads. Experiments on RLBench show that structured perception training boosts closed‑loop success, with XGenAct achieving 52% success on five external tasks compared to 26% for the best baselines, and it outperforms pipelines that first generate RGB and then apply a frozen perception expert for depth and segmentation prediction.
arXiv:2609.19142v1 Announce Type: new Abstract: World models endow perceptual systems with the ability to predict how scenes evolve under interaction. They are most beneficial when trained on diverse...
The paper introduces Spatially Aware World Action Model (SA‑WAM), a diffusion‑based framework that extends existing World Action Models by incorporating depth information alongside RGB to enable 3‑D‑aware action and future‑state prediction. SA‑WAM repurposes a pretrained video diffusion model, using a nonlinear encoding to map unbounded depth into the tokenizer’s bounded domain, thus preserving pretrained visual priors without 3‑D‑specific fine‑tuning. The model achieves state‑of‑the‑art performance on RoboCasa and LIBERO‑Plus benchmarks and demonstrates superior real‑world performance on a UR5 robotic arm in randomized environments, while also providing analysis linking world‑model prediction quality to rollout success.
StereoEngine introduces two methods for generating realistic synthetic stereo data for robotic manipulation. The first method warps a monocular video into a right view using depth estimation and a video diffusion model to inpaint disoccluded regions, while the second method renders stereo pairs directly in simulation. These approaches outperform existing baselines in stereo quality and enable improved bimanual manipulation policies when used for mid‑training or task‑specific fine‑tuning.
World models enable agents to perform forward rollout and planning without real-world interaction. However, their application in open-world embodied intelligence remains limited by the high cost of action annotations and the heterogeneity of action spaces across platforms.
World models endow perceptual systems with the ability to predict how scenes evolve under interaction. They are most beneficial when trained on diverse volumes of data, to instill a rich prior into do...
DeFM is a self‑supervised foundation model trained solely on depth images for robotics. It uses a DINO‑style self‑distillation objective on 60 M curated depth images to learn geometric and semantic representations that generalize across environments, tasks, and sensors. The model incorporates a novel input normalization strategy for metric awareness, can be distilled into compact versions for resource‑constrained systems, and achieves state‑of‑the‑art performance on depth‑based classification, segmentation, navigation, locomotion, and manipulation benchmarks, with strong sim‑to‑real transfer.
arXiv:2607. 16314v1 Announce Type: cross Abstract: World models, especially based on JEPA architectures, have been shown to learn robust dynamics of various environments.
arXiv:2503. 19947v2 Announce Type: replace-cross Abstract: Generalized metric depth understanding is critical for precise vision-guided robotics, which current state-of-the-art (SOTA) vision-encoders do not support.
arXiv:2608. 07267v1 Announce Type: new Abstract: Recent vision-language navigation (VLN) systems increasingly adapt pretrained vision-language models (VLMs) into vision-language-action (VLA) policies that map egocentric observations and language instructions directly to navigation actions.
arXiv:2607. 07885v1 Announce Type: cross Abstract: Dynamic obstacle avoidance in unstructured outdoor environments remains a critical challenge for autonomous mobile robots, particularly when large-scale robot-specific training data and simulation-based policies are impractical.