Auto-HSI is a system that creates personalized human‑swarm interaction interfaces on demand using large language models to automatically generate code from natural language descriptions and gesture demonstrations. The prototype enables untrained operators to control a swarm of 50 simulated robots with one‑ or two‑hand gestures, allowing teleoperation of motion, formation shape, and shape deformation. Experiments demonstrate the system’s gesture tracking, code generation, and live operation capabilities, including real‑time updates and deployment on real robots.
arXiv:2609.38982v1 Announce Type: cross Abstract: Coding agents powered by large language models (LLMs) have shown remarkable abilities to autonomously reason about and achieve goals in the digital w...
Coding agents powered by large language models (LLMs) have shown remarkable abilities to autonomously reason about and achieve goals in the digital world. However, bringing this success to the physica...
arXiv:2609.37089v1 Announce Type: new Abstract: Real-world videos provide rich demonstrations of manipulation, but turning them into reusable robot skills requires visually aligned environments, exec...
arXiv:2606. 18747v1 Announce Type: cross Abstract: Expressive gestures are essential for natural and effective communication, complementing speech when verbal cues alone are insufficient (e.
arXiv:2608. 15549v1 Announce Type: cross Abstract: Programming small social robots from natural-language instructions requires more than invoking isolated APIs.
RoboCousin is an extensible simulation platform that transforms user-provided observations into reusable assets, scenes, and expert trajectories for bimanual robotic manipulation. It converts object images into simulation-ready models with visual, collision, semantic, and physical metadata, automatically generates grasp candidates, and builds digital cousins that vary objects, backgrounds, layouts, and language instructions while preserving task-relevant affordances. The platform supports both tabletop and room-level scene construction, and the authors release RoboCousin-OBD with over 3,000 annotated objects and 50 backgrounds, generating more than one million expert trajectories across 50 tasks, with simulation and real-robot experiments demonstrating comparable annotation quality and effective sim-to-real transfer.
arXiv:2609.12541v1 Announce Type: new Abstract: We demonstrate that a general-purpose agent can directly drive a physical robot throughout task execution without any task-specific or environment-spec...
arXiv:2609.24660v2 Announce Type: cross Abstract: Human demonstrations offer a scalable way to collect manipulation data, but their contacts may be unstable or infeasible when transferred to a robot...
HIL-UMI is a policy-guided Universal Manipulation Interface that enables robot‑free, human‑in‑the‑loop post‑training of vision‑language‑action models. By querying the current policy during handheld demonstrations and using an Energy Score to detect out‑of‑distribution states, it selectively collects new data and refines a progress‑based advantage estimator. The updated estimator then drives advantage‑conditioned behavioral cloning, improving performance on long‑horizon and precise manipulation tasks while reducing per‑frame collection time compared to HG‑DAgger.
arXiv:2607. 23784v1 Announce Type: cross Abstract: While vision-language-action models have demonstrated impressive zero-shot manipulation capabilities, they remain fundamentally black box policies that are difficult to interpret, adapt, or correct when they inevitably fail.
arXiv:2607. 04434v1 Announce Type: cross Abstract: Generalist robot manipulation policies have advanced rapidly, yet existing benchmarks remain limited in systematically evaluating their capabilities.
arXiv:2602. 09153v2 Announce Type: replace-cross Abstract: Simulation has become a key tool for training and evaluating home robots at scale, yet existing environments fail to capture the diversity and physical complexity of real indoor spaces.