Robotics and embodied AI

Manipulation, locomotion, sim-to-real transfer and autonomous driving: learning systems that have to survive physics.

4,108 stories · RSS feed

arXiv Computer Vision
Sep 7

CoLMIN: LLM-based Multi-Decision Path Negotiation for Cooperative Autonomous Driving

CoLMIN is an LLM-based framework for cooperative autonomous driving that addresses premature convergence to suboptimal solutions in multi-solution traffic scenarios. It introduces a Multi-Intent Negotiation module that generates multiple candidate driving intentions, an Evaluation-based Shallow Reflection Module that provides feedback to accelerate consensus, and a Deep Reflection Module that mitigates cognitive fixation by reflecting on negotiation histories. Experiments in the CARLA simulation show that CoLMIN outperforms existing methods in challenging interactive driving scenarios.

By Zhe Huang, Zhaoxin Fan, Shuo Wang, Wenjun Wu, Xuan Zhao, Min Liu
arXiv Computer Vision
Sep 7

VoxelFix: Post-Hoc Semantic Correction of Completed 3D Voxel Maps

VoxelFix is a graph‑based post‑hoc semantic correction method that refines voxel labels in completed 3D voxel maps while preserving their geometry and occupancy. It learns to correct errors by exploiting local geometry and neighboring semantic information, using training pairs generated by corrupting annotated maps with class confusions from upstream perception pipelines. Experiments on OccuFly maps show consistent improvements of 4.23–5.00 percentage points in mIoU, especially for tree, roof, and wall classes, and the method generalizes to out‑of‑distribution aerial scenes.

By Sunesh Praveen Raja Sundarasami, Taehyoung Kim, Johannes Scherer, Toma\v{z} Coti\v{c}, Sivasubiramaniam Subbiah, Andreas Greiner, Paul Spannaus, Sebastian Houben
arXiv Computer Vision
Sep 7

LensStyle: Learning the Optical Aesthetics for Controllable Stylized Lens Effect Rendering

LensStyle is a unified framework for controllable stylized lens effect rendering that explicitly models lens aesthetics through joint continuous‑discrete control. It uses a Dual‑Path Controller to separate continuous optical parameter modulation (e.g., focus distance, blur strength) from discrete lens‑style conditioning (e.g., circular, polygonal, donut, cat‑eye, starburst effects), allowing fine‑grained, interpretable, and physically grounded manipulation. The authors also curate a MultiLens dataset of multi‑lens image pairs synthesized under real optical constraints, and experiments show LensStyle outperforms existing lens effect rendering methods and diffusion‑based image editing models in realism, controllability, and aesthetic quality.

By Yachuan Huang, Liwen Xiao, Liao Shen, Qiwen Wang, Huiqiang Sun, Zhiyu Pan, Zhiguo Cao
arXiv AI
Sep 7

RoboSPA: Can VLA Models Go Beyond Simple Scenes and Short-Horizon Tasks?

RoboSPA is a large-scale robotic manipulation dataset and benchmark designed to evaluate Vision‑Language‑Action models on fine‑grained spatial reasoning and long‑horizon procedural planning. It contains 10 task categories, 56 base tasks, and 280 variants across five difficulty levels, with 527K trajectories collected from multiple embodiments and scenes. The benchmark introduces diagnostic metrics beyond binary success, revealing that current VLA models struggle with complex spatial relations, precise execution, and memory‑intensive planning.

By Zhenxuan Fan, Bo Zhang, Yutong Lin, Yuqian Yuan, Juekai Lin, Liang Liang, Zhuoyi Huang, Wenqiao Zhang, Juncheng Li, Siliang Tang, Jun Xiao, Yueting Zhuang
arXiv AI
Sep 7

How do LLMs Evaluate Perceived Moral Agency? Investigating Moral Decision-Making in Human-Artificial Agents Interactions

The paper reports the first empirical study comparing how humans and large language models (LLMs) evaluate perceived moral agency (PMA) in both human and autonomous artificial agents within smart city scenarios. Using a validated PMA scale, 190 human participants and various LLMs were assessed, revealing that humans are perceived to have higher moral agency than artificial agents. When confronted with moral dilemmas, LLMs focus on situational factors such as harm severity and urgency, mirroring the context‑sensitivity observed in human raters.

By Fernanda Mansilla, Aloysius Tok, Bahia Guella\"i, Farah Benamara, Nancy F. Chen
arXiv AI
Sep 7

MultihopSpatial: Multi-hop Compositional Spatial Reasoning Benchmark for Vision-Language Model

MultihopSpatial is a new benchmark for Vision‑Language Models that focuses on multi‑hop, compositional spatial reasoning with queries ranging from 1 to 3 hops across varied spatial perspectives. It introduces the Acc@50IoU metric, which jointly evaluates answer selection and precise bounding‑box prediction, and provides a large‑scale training corpus, MultihopSpatial‑Train, to improve spatial intelligence. Evaluation of 37 state‑of‑the‑art VLMs shows that compositional spatial reasoning remains a significant challenge, and reinforcement learning fine‑tuning on the corpus boosts both intrinsic spatial reasoning and downstream embodied manipulation performance.

By Youngwan Lee, Soojin Jang, Yoorhim Cho, Seunghwan Lee, Yong-Ju Lee, Sung Ju Hwang
arXiv Computer Vision
Sep 7

CrossDepth: Geometry-Constrained Attention for Generalizable Multi-View Surround Depth Estimation

CrossDepth introduces geometry-constrained attention for multi-view surround depth estimation, addressing cross-image inconsistencies caused by varying camera intrinsics and limited receptive fields. The method conditions features on per-pixel camera-aware ray embeddings and extends pixel context via cross-image attention limited to geometrically plausible regions. Trained self-supervised with photometric consistency, it achieves better depth accuracy and consistency on DDAD and nuScenes compared to existing self-supervised approaches.

By Samer Abualhanud, Max Mehltretter
arXiv Computer Vision
Sep 7

SocioGesture: Real-Time and Adaptive Social Gesture Perception for Human-Robot Interaction

SocioGesture is a real‑time, adaptive system for recognizing social gestures in human‑robot interaction. It employs a compact, confidence‑aware body‑hand skeleton representation and a lightweight dual‑stream model that fuses body motion with hand articulation, enabling low‑latency onboard recognition. The model is trained with occlusion‑aware skeleton corruption to handle missing hands, occluded arms, and unstable keypoints, and it can expand its gesture vocabulary during deployment by saving uncertain interaction segments for offline labeling.

By Wenjin Fu, Li-Fan Wu, Jerin Peter, Chip Huyen, Boyuan Chen, Jan Liphardt
arXiv AI
Sep 7

A Schema Bounded Language Model for Refining Robot Policies Without Destabilizing Local Learning

The paper presents a decentralized navigation framework for composite heterogeneous robots that integrates a large language model (LLM) policy agent, an Upper Confidence Bound (UCB) bandit, and a Double Deep Q-Network (Double DQN) controller. Each robot independently generates and refines policies at the round level using LLM inference, while the Double DQN handles tick-level action selection based on navigation variables and LLM priors. Across 30 rounds, the full configuration achieved all goals with the lowest median completion time (42 ticks) and a 25–39% improvement over other setups.

By Chongwen Dong, Mithun Paul Saint-Germain, Pinjari Asif, Carlo R. daCunha
arXiv Computer Vision
Sep 7

E-RGB-D: Real-Time Event-Based Perception with Structured Light

The paper introduces E‑RGB‑D, a real‑time event‑based perception system that combines a Digital Light Processing projector with a monochrome event camera to produce RGB‑D data. By projecting structured light and capturing asynchronous brightness changes, the system can detect color and depth for each pixel, achieving a color detection speed of 1400 fps and a depth detection rate of 4 kHz. The approach enables frameless RGB‑D sensing and delivers colorful point clouds without compromising spatial resolution.

By Seyed Ehsan Marjani Bajestani, Giovanni Beltrame
arXiv AI
Sep 7

VLA-Precision: Asymmetric Co-Bootstrapping for Efficient Real-World Online RL of Vision-Language-Action Models

VLA-Precision introduces an efficient real‑world online reinforcement learning framework for vision‑language‑action (VLA) models, featuring the Asymmetric Co‑Bootstrapping (ACoB) algorithm and the ACoB‑Stream architecture. ACoB uses asymmetric co‑bootstrapping across timescales to rapidly improve policy performance while refining value estimates, thereby reducing policy drift. ACoB‑Stream enables large VLA models to run with up to 10.9× higher throughput and computational efficiency, achieving a 98.3 % mean success rate on nine high‑precision chemistry tasks in under 46 minutes per task.

By Chenyu Su, Zhaolong Shen, Yuan Qian, Chen Qian, Rui Zhang, Feng Yan, Weixing Chen, Fei Zhang, Jiamin Wang, Shuang Cong, Weiwei Shang
arXiv AI
Sep 7

Linguistic Trajectory Encoding for Efficient Long-Horizon Spatial Memory in Embodied Agents

The paper introduces Linguistic Trajectory Encoding (LTE), a hybrid representation that compresses dynamic object motion histories using natural language descriptions, sparse spatial anchors, and visual anchors. LTE adapts compression to motion complexity by anchoring periods without reliable observations to the last seen location while preserving accuracy with geometric waypoints and linguistic descriptions. Evaluated on the newly constructed Spatial Memory Benchmark (SMB) from EgoLife multi‑day recordings, LTE achieves 45.3 % success in semantic trajectory retrieval and 48.7 % in long‑horizon object retrieval, outperforming prior structured‑memory and VLM baselines, and compresses trajectories 8.7×–26.1× with sub‑second query latency on 24‑hour video.

By Tianyidan Xie, Shenyi Wang, Qiang Tang, Mingjie Wang, Zhicheng Qiu, Xuanfu Li, Zhan Xu, Jian Yang, Lanjun Wang, Zili Yi
arXiv Computer Vision
Sep 7

MINT: A Unified Model for World-Space Camera and Hand Motion Estimation from Scalable Egocentric Pipeline Supervision

MINT is a foundation model that directly predicts world-space two-hand trajectories from egocentric RGB video, jointly estimating camera motion, hand states, and hand presence in a single spatiotemporal representation. It uses an open-source labeling pipeline, EGOPIPELINE, to generate large-scale pseudo-labels for pretraining, followed by fine-tuning on a small set of high-quality joint annotations. The model outperforms existing multi-stage approaches in accuracy and speed, and generalizes zero‑shot to unseen egocentric datasets.

By Zijie Zhu, Weiren Cai, Yizhou Wang, Zhenjie Yang, Yide Liu, Jiahao Chen, Guanqi He
arXiv AI
Sep 7

Cost-Aware Hierarchical Multi-Agent Ransomware Detection and Family Attribution

The paper introduces a Cost-Aware Hierarchical Multi-Agent System (HMAS) for ransomware detection and family attribution that adaptively selects analysis modalities to balance accuracy and computational cost. Static analysis is used first, with dynamic and memory modalities added only when confidence is low or specialist agents disagree, guided by a cost model. Experiments show HMAS achieves high accuracy (96.57% binary detection, 0.90 macro‑F1 attribution) while reducing analysis cost by 43.97% and latency, with 56.05% of cases resolved using static evidence alone.

By Mubashar Iqbal, Asifullah Khan
Hugging Face Trending Papers
Sep 4

CoLMIN: LLM-based Multi-Decision Path Negotiation for Cooperative Autonomous Driving

CoLMIN is an LLM-based framework for multi-decision path negotiation in cooperative autonomous driving. It introduces a Multi-Intent Negotiation module that generates multiple driving intentions, an Evaluation-based Shallow Reflection Module that provides feedback to accelerate consensus, and a Deep Reflection Module that mitigates cognitive fixation by reflecting on negotiation histories. Experiments in the CARLA simulation show that CoLMIN outperforms existing methods in challenging interactive driving scenarios.

arXiv Computer Vision
Sep 4

RoboTok: An Internet-Scale Data Engine for Human Demonstration Retrieval and Dexterous Manipulation Learning

RoboTok is an internet‑scale data engine that retrieves human manipulation videos from the web to train dexterous robot policies. It learns a latent motion space from 3D hand trajectories in actor‑centered reference frames, allowing manipulation behaviors to be compared across different viewpoints, scenes, and occlusions while remaining compact for efficient search. Experiments show RoboTok retrieves more relevant demonstrations and improves downstream robot task success compared to existing retrieval methods.

By Howard Qian, Yiting Chen, Yunfei Xie, Kejia Ren, Podshara Chanrungmaneekul, Gaotian Wang, Bowen Wen, Chen Wei, Kaiyu Hang
arXiv AI
Sep 4

Rethinking World Models for Safety-Critical Embodied Systems

The article discusses how current world models, while achieving high predictive likelihood and visual fidelity, often fail to preserve the evidence needed for safe decision-making in embodied systems. It identifies three structural mismatches—likelihood versus risk, prediction versus intervention, and finite-horizon prediction versus accumulated consequences—and proposes the Risk‑Informed World Model (RIWM) as a decision‑centric framework. RIWM emphasizes consequences, intervention, epistemic uncertainty, and recoverability, integrating decision‑relevant representation, counterfactual reasoning, safety‑critical episodic memory, and runtime safety assurance to better support safety‑critical embodied systems.

By Kailang Ma, Heye Huang, Inhi Kim, Kitae Jang
arXiv AI
Sep 4

Artificial Intelligence for Energy Optimization in Data Centers

The paper reviews 194 papers on using artificial intelligence to optimize data center energy use, coding 63 of them. It finds that most control studies validate only in simulation, none consider water withdrawal or embodied carbon, and savings estimates overlap across methods, preventing ranking. The authors propose CLEAR‑DC, a framework that links control and workload demand through elasticity, reports net benefits, and records energy, carbon, water, embodied share, and validation venue.

By Mohammed Basharath Ullah, Summaiya Unnisa Begum, Mohammed Nadeem Ullah
arXiv AI
Sep 4

Decentralized Vision-Based Autonomous Aerial Wildlife Monitoring

The paper introduces a decentralized, vision-based system using multiple quadrotors equipped with a single RGB camera for monitoring wildlife. It emphasizes scalability, low bandwidth, and minimal sensor requirements, enabling robust identification and tracking of large species in natural habitats. The authors present novel coordination and tracking algorithms that operate without centralized communication, and validate the approach with real-world field experiments.

By Makram Chahine, William Yang, Alaa Maalouf, Justin Siriska, Ninad Jadhav, Daniel Vogt, Stephanie Gil, Robert Wood, Daniela Rus
arXiv AI
Sep 4

FailBench: How Reliable are VLMs at Judging Robot Task Success?

FailBench is a new benchmark for robot failure detection, containing 2,197 manipulation attempts from 14 public sources, with 75% of failures occurring naturally. The study evaluates 13 vision‑language model (VLM) detectors, finding the best model achieves only 0.77 mean balanced accuracy, and that fine‑tuned failure detectors often underperform general‑purpose VLMs. Performance varies with visual evidence, excelling when object motion is observable but dropping to near chance on contact‑intensive assembly tasks, and input‑level cropping of outcome‑relevant regions improves the top detector by 2.4 percentage points.

By Zaruhi Navasardyan, Tatul Danielyan, Hrant Davtyan