Splat2Act: From Captured Scenes to Personalized VLA Manipulation

Splat2Act is an automated real‑to‑sim‑to‑real pipeline that personalizes a pretrained vision‑language‑action policy for a specific indoor scene using only a handheld phone scan and images of the user’s objects. It constructs a photorealistic digital twin with 3D Gaussian splatting, discovers receptacles, generates over 68,000 pickup demonstrations via a simulation oracle, and fine‑tunes the policy in two stages—first on public objects, then on the user’s own items—achieving up to 62.5% pickup success in target scenes. The approach demonstrates that such personalization can be achieved without costly real‑world demonstrations, with a real Stretch 3 robot matching simulated success rates on a 12‑task subset.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computer Vision.

• robotics
• fine-tuning
• multimodal