Robotics and embodied AI

Manipulation, locomotion, sim-to-real transfer and autonomous driving: learning systems that have to survive physics.

4,108 stories · RSS feed

arXiv Computer Vision
Sep 3

MultiGraspNet: A Multitask 3D Vision Model for Multi-gripper Robotic Grasping

MultiGraspNet is a multitask 3D vision model that simultaneously predicts feasible poses for both parallel and vacuum grippers, allowing a single robot to handle multiple end effectors. Trained on the aligned GraspNet-1Billion and SuctionNet-1Billion datasets, it generates graspability masks that quantify the suitability of each scene point for successful grasps. With only 15.75 M parameters, the model achieves fast inference on a single GPU and demonstrates competitive performance against single-task models while reducing computational cost, as shown in extensive experiments and real‑world tests on a single‑arm multi‑gripper setup.

By Stephany Ortuno-Chanelo, Paolo Rabino, Enrico Civitelli, Tatiana Tommasi, Raffaello Camoriano
arXiv Machine Learning
Sep 3

Sim2Signal: Sim-to-Real Benchmarks for Traffic Signal Control

Sim2Signal is a benchmark designed to systematically measure the Sim-to-Real gap in traffic signal control by decomposing it into observation, action, transition, and reward gaps. The study evaluates 18 mitigation methods across 33 gap settings and 10 calibrated networks from five real-world locations, finding that direct transfer degrades performance but mitigation effectiveness varies by network and gap type. The most effective approaches tend to estimate the specific changes caused by each gap rather than relying on domain randomization or invariant representations.

By Ferdous Al Rafi, Susrik Mukherjee, Latika Liladhar Dekate, Jennifer Yawa Lavoe, Huaiyuan Yao, Shlok Mohanty, Longchao Da, Xuesong Zhou, Hua Wei
arXiv AI
Sep 3

Epistemic Sybil Resistance: Multiplying AI Agents Without Multiplying Evidence

The paper introduces the concept of an epistemic Sybil problem in multi‑agent AI systems, where multiple agents may produce seemingly independent reports that actually stem from the same underlying evidence. It formalizes this issue using information‑theoretic measures and demonstrates through large‑scale experiments that naive aggregation of replicated reports can severely degrade inference accuracy unless the system accounts for shared evidence ancestry and correlated extraction errors. The study shows that aggregators that track evidential dependence rather than merely report multiplicity or similarity achieve better calibration and inference performance.

By Marc Bara
arXiv Computer Vision
Sep 3

DailyBench: A Unified Benchmark for AI-Generated and Manipulated Images from Modern Generative Models

DailyBench is a unified benchmark designed to evaluate AI-generated image detectors on both modern full-image synthesis and object-level manipulation. It comprises two subsets: FakeBench, featuring high‑quality images from recent open‑source and commercial generative models, and ManipulationBench, containing subtle local edits applied to real images using advanced image‑conditional models. Experiments show that detectors with high accuracy on older datasets perform poorly on DailyBench, revealing significant robustness gaps.

By Xin Jiang, Hao Tang, Junyao Gao, Meiqi Cao, Fei Shen, Dongming Zhang, Yongdong Zhang
arXiv Machine Learning
Sep 3

DiDrive: A Risk-Aware Hierarchical Diffusion Framework for Safe Offline Reinforcement Learning in Autonomous Driving

DiDrive introduces a risk‑aware hierarchical diffusion framework for offline reinforcement learning in autonomous driving. It combines a low‑level risk‑gated encoder with a high‑level contextual modulator to filter redundant state information, and a 3DICE policy optimization that reduces out‑of‑distribution overestimation and stabilizes gradients. On the CARLA benchmark, DiDrive outperforms baselines such as IQL, CQL, and Diffusion‑QL, achieving an 85% success rate and a 4295.68 average reward in dense traffic with 60 vehicles.

By Qisong Guo, Jingtang Chen, Zhilin Chen, Pei Xu, Mingjian Fu, Wenxi Liu, Yuanlong Yu
arXiv Computer Vision
Sep 3

Towards Open-World Referring Expression Comprehension: A Benchmark with Training-free Multi-task Consistency Checker

The paper introduces OpenRef, a benchmark for Referring Expression Comprehension (REC) designed for open‑world scenarios. OpenRef expands beyond simple settings by including diverse visual domains, variable target counts (multi‑target and none‑target), and a rich vocabulary with proper nouns, polysemous words, and ordinal terms. It also proposes new evaluation metrics—F1 for grounding accuracy and N3R for negative expression rejection—and presents a training‑free Multi‑task Consistency Checker (MCC) that improves model performance with a single click.

By Zongjian Wu, Lei Zhang
arXiv Computer Vision
Sep 3

SignMatch: Matching Dictionary Signs to Continuous Sign Language Video

SignMatch introduces a prototype‑structured embedding space that learns to match dictionary sign videos with continuous sign language footage based solely on visual similarity of handshape and motion. By mapping isolated dictionary exemplars into this space, the method enables direct, embedding‑based sign matching and can generalise to unseen signs using only dictionary examples. Experiments on ASL‑Citizen, ChaLearn OSLWL, and BOBSL CSLR2 benchmarks show strong cross‑dataset, cross‑task, and cross‑language performance, outperforming prior approaches on American, British, and Spanish sign languages without benchmark‑specific supervision.

By Ryan Wong, Youngjoon Jang, Liliane Momeni, G\"ul Varol, Andrew Zisserman
arXiv AI
Sep 3

CHASE: Cache-Hole-Adapted Skip Exit for Looped State-Space Language Models

The paper introduces CHASE, a cache‑hole‑adapted skip‑exit mechanism for looped state‑space language models, specifically Looped Mamba and Looped Hybrid Mamba‑Transformer. It shows that looping these architectures improves performance on controlled reasoning tasks and remains competitive in pre‑training benchmarks while using fewer distinct parameters. The cache‑hole adaptation allows selective skipping of recurrent steps during inference, maintaining perplexity close to full computation and achieving significant speedups.

By Zhenxuan Yu, Takeshi Kojima, Yutaka Matsuo, Yusuke Iwasawa
arXiv Computer Vision
Sep 3

Geometry-Guided Modeling of Foundation Features Enables Generalizable Object Shape Deformation Learning

The paper introduces a generalizable deformation learning framework that reconstructs 3D objects by deforming a category-level shape template to match a monocular observation. It employs a geometry-guided feature modeling mechanism to enrich foundation features with template topology, creating a geometry-aware representation that is explicitly correlated with the target observation for precise deformation. A view-adaptive feature aggregation module further bridges the gap between the fixed template and arbitrary target views by leveraging multi-view template features and camera poses, ensuring robust feature alignment across diverse viewpoints.

By Yiyao Ma, Kai Chen, Zhongxiang Zhou, Zhuheng Song, Dongsheng Xie, Zelong Tan, Rong Xiong, Qi Dou
arXiv Machine Learning
Sep 3

Tri-Band Channel Measurement-Enabled Multi-Layer Digital Twin for Terahertz Wireless Data Centers

The paper proposes a measurement-driven multi-layer digital twin framework for terahertz (THz) wireless data centers. It begins with extensive tri-band channel measurements at 140, 220, and 300 GHz to calibrate a physical twin that optimizes geometry, material, antenna, and propagation models. An AI channel twin, built on a line-of-sight aware implicit neural field, learns location-dependent channel statistics to enable real‑time prediction of received power and LoS probability, which feeds into a system‑level evaluation layer that analyzes coverage and interference for AP‑to‑rack and rack‑to‑rack links. Experimental results show the AI twin achieves lower power reconstruction error than existing neural‑field baselines while maintaining real‑time inference, and ceiling‑mounted AP deployment yields over 90% coverage at a 10 dB SINR threshold.

By Mingjie Zhu, Ziming Yu, Guangjian Wang, Chong Han
arXiv AI
Sep 3

What Drives Success in Physical Planning with Joint-Embedding Predictive World Models?

The paper investigates Joint-Embedding Predictive World Models (JEPA-WMs), a class of methods that perform planning in a learned representation space rather than raw input space. It systematically studies how model architecture, training objectives, and planning algorithms influence success across simulated and real‑world robotic tasks, and proposes a JEPA-WM variant that surpasses established baselines in navigation and manipulation. The authors provide code, data, and checkpoints for reproducibility.

By Basile Terver, Tsung-Yen Yang, Jean Ponce, Adrien Bardes, Yann LeCun
arXiv AI
Sep 3

Dictionary-Guided Mutation Operators for Automated HDL Repair

The paper introduces a dictionary-guided HDL repair system that uses ANTLR-derived mutation vocabularies and a simulation-divergence fault localization module to generate syntactically valid Verilog mutations. The mutation operator performs token substitutions, insertions, and deletions via regex matching, while the fault localization scores source lines based on proximity to diverging output wires, guiding the search. Evaluated on the CirFix benchmark, the approach achieves correct repairs on 14 bug variants, including a multi-bug case, and outperforms CirFix with an 18x speedup on a two-edit benchmark.

By Maisha Mastora, Dean Sullivan
arXiv AI
Sep 3

SkillGLoW: Procedural-Family Skill Consolidation for Self-Improving Agents on Long-Horizon Task Streams

SkillGLoW introduces a new way for large language model agents to self‑improve by consolidating procedural skills shared across related tasks. Instead of storing all skills in a single global document or a flat per‑task pool, SkillGLoW aggregates local skills into procedural families, compresses them into de‑instantiated global priors, and regenerates instance‑specific details on demand. Experiments on four diverse benchmarks show that these priors improve performance by an average of 17.2 points over a no‑skill baseline, are more compact than per‑task pools, and enable better transfer to unseen tasks.

By Ao Yan, Xin Zhang, Jiawei Du, Joey Tianyi Zhou
arXiv Computer Vision
Sep 3

Glass Segmentation with Fusion of Learned and General Visual Features

The paper introduces a dual‑backbone architecture for glass segmentation that combines a frozen foundation model with a learned backbone trained on glass‑specific data. By fusing hierarchical multi‑scale features from both backbones, the method produces accurate segmentation masks and achieves state‑of‑the‑art performance on four benchmark datasets. Ablation studies confirm the benefits of the dual‑backbone design and its generalizability across different backbone choices, while also offering competitive inference speeds, especially with lighter backbones.

By Risto Ojala, Tristan Ellison, Mo Chen
arXiv AI
Sep 3

DiffuSearch: How Hybrid Trajectory Planning Benefits from Aligned Objectives in Diffusion and Action Space

DiffuSearch is a hybrid trajectory planner for autonomous driving that unifies objectives across both generation and refinement stages. It first uses a guided diffusion model to produce scene-consistent joint trajectories, then refines them with a Monte Carlo Tree Search that shares the same driving goals—collision avoidance, drivable area compliance, comfort, and progress. Experiments on nuPlan and interPlan benchmarks show that this synergy reduces collisions and improves comfort, especially in complex interactive scenarios.

By Steffen Hagedorn, Aron Distelzweig, Alexandru P. Condurache
arXiv Computer Vision
Sep 3

VIPS: Vehicle-Infrastructure Cooperative Planning Benchmark via Pseudo-Simulation

VIPS is a benchmark for vehicle‑to‑infrastructure cooperative autonomous driving that uses pseudo‑simulation to combine vehicle and infrastructure observations, enabling scalable yet realistic evaluation of robustness and error propagation without full simulation. The paper also introduces CoS‑V2X, a cooperative planning framework that employs sparse representations to model vehicle‑infrastructure interactions efficiently and robustly under heterogeneous observations.

By Hoonhee Cho, Jae-Young Kang, Giwon Lee, Hyemin Yang, Heejun Park, Kuk-Jin Yoon
arXiv Computer Vision
Sep 3

Towards Zero-Shot Transfer Across Embodiments For Driving VLAs

The paper investigates how Vision‑Language‑Action (VLA) models can generalise across different driving environments and camera setups. It introduces a multi‑dataset training strategy and an auxiliary objective called BEV‑Forcing, which injects bird‑eye‑view spatial information into the VLA backbone to improve both in‑distribution and out‑of‑distribution performance on a limited number of camera rigs. The authors observe that while BEV‑Forcing helps when training data is scarce, its advantage diminishes as the number of training embodiments grows, suggesting that scaling diversity may reduce the impact of such auxiliary tasks.

By Caio Azevedo, Stefano Sabatini, Sascha Hornauer, Fabien Moutarde
arXiv Computer Vision
Sep 3

Make-It-Poseable: Feed-forward Latent Posing Model for 3D Characters

Make‑It‑Poseable is a feed‑forward framework that treats 3D character posing as a skinning‑free latent‑space transformation. It decouples shape deformation from fixed mesh connectivity, using a latent posing transformer, dense pose representation, and an adaptive completion module with bipartite‑matched latent loss. Experiments show it outperforms existing baselines, generalizes to varied morphologies, and supports 3D authoring tasks such as part replacement and refinement.

By Zhiyang Guo, Ori Zhang, Jax Xiang, Alan Zhao, Zhenxun Yuan, Wengang Zhou, Houqiang Li
arXiv Computation and Language
Sep 3

How LLMs Build Fictional Worlds: Setting and Narrative Space in AI-Generated Creative Storytelling

The paper investigates how Large Language Models (LLMs) construct fictional worlds, specifically examining setting as a measurable aspect of storyworld creation. By generating 1,000 AI stories per model in English and German and comparing them to human-authored fiction from Project Gutenberg, the authors classify narrative space into five categories—action, perceived, visual, descriptive, and no space—using fine‑tuned BERT classifiers. Results show that human texts mainly use action space, grounding narratives in character-environment interaction, while LLMs consistently overproduce perceived space, focusing on atmosphere and affect, with this pattern varying by model and language.

By Katrin Rohrbacher, Bj\"orn Nieth, Emmanuelle Salin, Bjoern Eskofier, Michaela Mahlberg
arXiv Computation and Language
Sep 3

ShallowStream: Index Shallow then Answer Deep for Streaming Video Understanding

ShallowStream is a framework for streaming video understanding that uses the shallow layers of a multimodal large language model (MLLM) to encode frames and build a lightweight index. During streaming, it maintains an always‑on index via the KV cache of shallow layers, and at query time it scores context frames using shallow‑layer attention and selects diverse evidence for answering. The approach matches the performance of leading streaming methods while cutting per‑frame prefill latency and 10‑second end‑to‑end latency by up to 52.1× and 11.9×, respectively.

By Jitai Hao, Ke Yang, Qiang Huang, Jun Yu