Robotics and embodied AI

Manipulation, locomotion, sim-to-real transfer and autonomous driving: learning systems that have to survive physics.

4,111 stories · RSS feed

arXiv AI
Aug 28

MOMO: A framework for seamless physical, verbal, and graphical robot skill learning and adaptation

The paper introduces MOMO, a framework that allows industrial robots to be adapted by non-experts using kinesthetic touch, natural language, and a graphical web interface. It combines energy‑based intention detection, a tool‑based LLM for safe language adaptation, Kernelized Movement Primitives for motion encoding, probabilistic Virtual Fixtures for guided demonstrations, and ergodic control for surface finishing. The authors validate the system on a 7‑DoF torque‑controlled robot at the Automatica 2025 trade fair, showing its practical applicability in industrial settings.

By Markus Knauer, Edoardo Fiorini, Maximilian M\"uhlbauer, Stefan Schneyer, Promwat Angsuratanawech, Florian Samuel Lay, Timo Bachmann, Samuel Bustamante, Korbinian Nottensteiner, Freek Stulp, Alin Albu-Sch\"affer, Jo\~ao Silv\'erio, Thomas Eiband
arXiv AI
Aug 28

Egosurg: Arbitrary view synthesis for egocentric replay of operating room workflows from ambient cameras

EgoSurg is a framework that reconstructs dynamic operating room scenes from sparse wall‑mounted stereo video and renders arbitrary, role‑specific egocentric views without instrumenting personnel. It builds a per‑timestamp 3D Gaussian Splatting representation using scale‑aware stereo depth and refines it with an image‑conditioned diffusion model to correct artifacts from limited camera coverage, crowding, and occlusion. Evaluations on real robotic pulmonology procedures and simulated sessions show consistent near‑field reconstruction fidelity (PSNR 26.8 dB, SSIM 0.895) and synthesized egocentric view quality (PSNR 17.8 dB, SSIM 0.766) across workflow phases and hospital sites, with case studies demonstrating applications such as sterile field violation adjudication, role‑specific replay, and counterfactual personnel positioning.

By Han Zhang, Lalithkumar Seenivasan, Jose L. Porras, Roger D. Soberanis-Mukul, Hao Ding, Hongchao Shu, Benjamin D. Killeen, Ankita Ghosh, Lonny Yarmus, Jeffrey K. Jopling, Masaru Ishii, Angela C. Argento, Mathias Unberath
arXiv Machine Learning
Aug 28

Making Latent Evolution Explicit: Operator-Structured Transitions for World Action Models

The paper introduces the Latent Evolution Operator Network (LEON), a new architecture for modeling latent state transitions in World Action Models (WAMs). LEON uses a learned observable space with context‑modulated operator‑based propagation and additive forcing, grounded in a controlled Koopman generator framework. Experiments on two WAM formulations show that LEON improves closed‑loop performance and robustness, demonstrating that the way transitions are realized is a crucial architectural choice in latent WAMs.

By Xiaoxiao Lu, Yunlong Dong, Jiahao Shi, Ye Yuan
arXiv Computer Vision
Aug 28

TrapVLA: Trapping Vision-Language-Action Models in Configured Failure Modes

The paper introduces Configured Failure Trapping, a new backdoor attack targeting Vision‑Language‑Action (VLA) models that activates through subtle textual triggers and forces the robot to fail in a specific, controlled manner. It presents a data engine for generating high‑quality target trajectories, an automated evaluation suite, and two benchmarks—Trap‑LIBERO and Trap‑RoboTwin—covering four failure modes. The authors propose TrapVLA, a method that learns trigger‑induced action residuals to steer policies toward the desired failure behavior, demonstrating effectiveness in both simulation and real‑world robotic experiments while maintaining performance on clean data.

By Jun-Hui Liu, Kun-Yu Lin, Yi-Lin Wei, Xu-Han Chen, Yinghao Li, Zhuohao Li, Yuan-Ming Li, Qing Zhang, Xiaoyi Fan, Dongmei Jiang, Yan Li, Wei-Shi Zheng
arXiv Machine Learning
Aug 28

STITCH-OPE: Trajectory Stitching with Guided Diffusion for Off-Policy Evaluation

STITCH-OPE is a model‑based generative framework that uses denoising diffusion to perform off‑policy evaluation (OPE) in high‑dimensional, long‑horizon settings. It generates synthetic trajectories for a target policy by guiding a diffusion model trained on behavior data, subtracting the behavior policy’s score to avoid over‑regularization and stitching partial trajectories to extend horizon length. The authors provide theoretical variance‑reduction guarantees and demonstrate improved mean squared error, correlation, and regret on D4RL and OpenAI Gym benchmarks.

By Hossein Goli, Michael Gimelfarb, Nathan Samuel de Lara, Haruki Nishimura, Masha Itkina, Florian Shkurti
arXiv Computer Vision
Aug 28

Think3D: Thinking with Space for Spatial Reasoning

Think3D introduces a framework that endows Vision‑Language Models with interactive 3D chain‑of‑thought reasoning by integrating 3D manipulation tools for active spatial exploration. The approach improves performance on benchmarks such as BLINK Multi‑view, MindCube‑1K, and VSI‑Bench‑Tiny for proprietary models like GPT‑4.1 and Gemini 2.5 Pro, and a reinforcement‑learning variant, Think3D‑RL, enables open‑weight models such as Qwen3‑VL‑4B to autonomously learn effective 3D exploration strategies, yielding tool‑use patterns comparable to stronger models and turning a performance drop on MindCube‑1K into a substantial improvement.

By Zaibin Zhang, Yuhan Wu, Lianjie Jia, Yifan Wang, Zhongbo Zhang, Yijiang Li, Binghao Ran, Fuxi Zhang, Zhuohan Sun, Yizhuang Peng, Zhenfei Yin, Lijun Wang, Huchuan Lu
arXiv AI
Aug 28

From Sound to Symptom: Real-Time Respiratory Signal Understanding for Conversational Healthcare Agents

The paper introduces HealthCUES, a real‑time streaming pipeline that extracts and analyzes cough and throat‑clearing events from live spoken conversations. It detects coughs within sub‑second latency, distinguishes cough subtypes (dry, wet, barking, whooping), differentiates coughing from throat clearing, and estimates temporal boundaries, all while gating alerts based on conversational context. The system, built on Qwen3Omni, achieves high accuracy (93% F1 for cough detection) and low latency (340 ms) and has been validated by healthcare professionals for telehealth use.

By Tanmay Laud, Herprit Mahal, Subhabrata Mukherjee
arXiv AI
Aug 28

CLAP: Cross-Embodiment Video World Models are Zero-Shot Physical Simulators

CLAP is a cross-embodiment framework for action‑conditioned video generation that can be trained on diverse internet‑scale videos from both humans and robots. It reconciles different action spaces—end‑effector poses, language instructions, and latent actions—using a curriculum that first learns physics priors from unlabeled video and then grounds them in real‑world action spaces for zero‑shot deployment. The resulting models match or exceed state‑of‑the‑art single‑embodiment models in challenging environments and support few‑shot adaptation across a wide range of robot morphologies.

By Kechen Liu, Ola Shorinwa
arXiv AI
Aug 28

Decoupling Planning and Control for Instructable Agents

The paper introduces Instruct-to-Act, a system that decouples planning and control by combining a vision‑language model (VLM) planner with a world‑model controller. The VLM generates sparse, high‑level text instructions, while the controller executes them at high frequency, trained via relabeling rollouts with synthetic instructions and joint optimization of behavior cloning, reward, and world‑model objectives. Across seven embodied environments—including multi‑agent settings—this approach outperforms controller‑only and direct VLM action methods, maintains fast control, and allows swapping pretrained VLM planners without fine‑tuning, achieving competitive results with strong baselines on most tasks.

By Zineng Tang, Kelsey R. Allen, Sjoerd van Steenkiste, Ishita Dasgupta, Alane Suhr
arXiv AI
Aug 28

A Unified Conditional Flow for Motion Generation, Editing, and Intra-Structural Retargeting

The paper introduces a unified conditional-flow framework that integrates text-driven motion generation, semantic editing, and intra-structural retargeting into a single rectified-flow model. By treating editing as a change in semantic condition and retargeting as a change in skeletal condition, the approach eliminates fragmented pipelines and allows a single model to perform generation, zero‑shot editing, and zero‑shot retargeting on articulated 3D motion data. Experiments on SnapMoGen and a Mixamo subset demonstrate that the model can handle all three tasks without task‑specific fine‑tuning, preserving both motion semantics and skeletal structure.

By Junlin Li, Xinhao Song, Siqi Wang, Haibin Huang, Yili Zhao
arXiv AI
Aug 28

Can You Say This for Me? Speaking Up by Proxy in Co-Located Discussion

SecondVoice is a mixed‑reality system that lets participants speak up in co‑located discussions via an embodied virtual proxy, separating the content of a point from the speaker’s identity. Users input their intent through a structured process, and the system reformulates and vocalizes it into the conversation. In a study with 16 participants, the proxy channel led to more points being voiced than an anonymous text board and sparked multi‑turn engagement that the text board did not elicit.

By Yue Shen, Rehema Abulikemu, Ryan P. McMahan, Yan Chen
arXiv AI
Aug 28

Why did My Robot Just Change Personality? Prompting Guidelines for a Grounded Robot Persona in LLM-Based HRI

The paper "Why did My Robot Just Change Personality? Prompting Guidelines for a Grounded Robot Persona in LLM-Based HRI" addresses the lack of clear prompt design in human‑robot interaction using large language models. It proposes a framework and a structured prompt template with eight functional components to specify, bound, and adapt robot behavior. The authors base their guidelines on a review of prior work and on survey data from 27 HRI experts, highlighting issues such as unclear robot personality, the need for user adaptation, and ethical concerns around safety, deception, and governance.

By Ashita Ashok, Franziska Babel, Patrick Holthaus, Rucha Khot, Karla Bransky, Fethiye Irmak Dogan, Karsten Berns, Silvia Rossi, Minha Lee, Guy Laban
arXiv Computer Vision
Aug 28

SimWAM: A Simple World Action Model for End-to-End Autonomous Driving

SimWAM is a lightweight World-Action Model that uses future‑video prediction only during training to supervise an action expert, enabling end‑to‑end autonomous driving without costly test‑time future imagination. The architecture co‑trains a pretrained video expert and a lightweight action expert via joint flow matching, while an isolated attention mask keeps action prediction independent of future frames. This design allows the video backbone to be swapped and the action expert scaled independently, achieving 91.5 PDMS on NAVSIM, outperforming state‑of‑the‑art WAM planners with lower latency and zero‑shot transfer to nuScenes.

By Zongchuang Zhao, Xin Zhou, Tianyang Xu, Zhengyang Sun, Kaixuan Zhou, Yu Wu, Honglin Li, Dingkang Liang, Xiang Bai
arXiv Computer Vision
Aug 28

Reconstructing Humans and Objects in Interaction using Large Reconstruction Models

The paper introduces MILO, a framework that uses Large Reconstruction Models (LRMs) to reconstruct detailed 3D human‑object interactions from a single image. By treating the LRM mesh as a geometric scaffold, MILO segments it into human and object parts, fits a parametric body model to the human component, and optionally aligns an object template to the object component. The approach achieves higher reconstruction accuracy than existing baselines across multiple benchmarks and interaction scenarios.

By Agniv Chatterjee, Georgios Pavlakos
arXiv Computer Vision
Aug 28

4DSynth: Controllable Procedural World Synthesis for Dynamic Embodied Simulation

4DSynth is a controllable procedural system that transforms natural-language descriptions, blueprint masks, or single photographs into editable 4D environments featuring explicit geometry, animated actors, collision-free trajectories, and physics-ready simulation states. The system unifies animation, camera planning, rendering, and task generation within a single geometry-grounded representation, enabling scalable creation of dynamic embodied simulation scenes. Using 4DSynth, the authors built 4DSynth-Nav, an interactive navigation benchmark that demonstrates the reproducibility and tunability of procedural failures across vision‑language models.

By Zehao Qi, Haochen Luo, Jia-Wang Bian, Zeyu Ma, Shuyang Sun
arXiv Computer Vision
Aug 28

Knowledge Distillation Driven Semantic NOMA with GAN Refinement for 6G Robotic Vehicle Networks

The paper introduces KDG‑SemNOMA, a framework for 6G robotic vehicle networks that combines knowledge distillation and generative models to improve semantic communication over uplink non‑orthogonal multiple access (NOMA). It employs a ConvNeXt‑based deep joint source‑channel coding architecture with an enhanced attention feature module for dynamic channel adaptation, and uses an orthogonal teacher model to guide a NOMA student model via two‑stage knowledge distillation. A channel‑conditional GAN further refines the reconstructed images, yielding higher pixel‑level accuracy and perceptual fidelity on the FFHQ‑256 dataset compared to state‑of‑the‑art methods.

By Qifei Wang, Zhen Gao, Li Qiao, Ziwei Wan, De Mi, Dapeng Li, Ying Sun
arXiv AI
Aug 28

Active sensing to characterize the heterogeneity of plant stress

The paper introduces an autonomous robotic platform that performs targeted chlorophyll fluorescence measurements on plant leaves. It integrates 3D plant reconstruction, geometric analysis, and motion planning to identify suitable leaf surfaces and generate collision‑free trajectories for a robotic manipulator. This system enables automated, repeatable, and spatially resolved physiological measurements that extend beyond passive imaging.

By Ayman Laaroussi, Peter Hanappe, David Colliaux
arXiv Computer Vision
Aug 28

Surgical Video Generation From Diffusion to World Models: A Survey

This survey reviews recent advances in surgical video generation, categorizing methods into unconditional, conditional, and world modeling generation. It highlights a shift from creating visually plausible frames to modeling the causal dynamics of surgical scenes, and discusses challenges such as pixel-level fidelity versus clinical plausibility, generalization, physical realism, controllability, and interpretability. The paper also compiles experimental results from public datasets to serve as a quantitative benchmark for the field.

By Fuxiang Huang, Chenxu Zhang, Liang Han, Lei Zhang
arXiv Computer Vision
Aug 28

DPA-I2P: Depth-Guided Projective Alignment for Image-to-Point-Cloud Registration in Autonomous Driving

The paper introduces DPA-I2P, a depth-guided projective alignment method for image-to-point-cloud registration in autonomous driving. It employs Ray-Conditioned Metric Depth Encoding and Projection-Consistent Vision Lifting to align depth and visual cues geometrically, and uses Cross-Modal Query Pruning to enhance matching stability. Experiments on KITTI and nuScenes show significant reductions in rotation and translation errors compared to existing implicit baselines.

By Wenxin Zhang, Hang Li, Zhiwei Xu, Qiankun Dong, Gang Wang, Tao Li
arXiv Computation and Language
Aug 28

DeflectBench: A Benchmark for Evaluating Rhetorical Fallacy Generation in LLMs

DeflectBench is a new benchmark that evaluates how large language models (LLMs) generate rhetorical fallacies when prompted. The study tests 23,990 generations from four leading models using three deflection strategies (whataboutism, ad hominem, red herring), seven prompt framings, and 80 claims across four controversy levels. Results show that refusal to produce fallacies depends mainly on request structure, with prompt framing and fallacy type dramatically affecting compliance rates.

By Art Kanke