The paper introduces a new benchmark for evaluating holistic, semantics-aware co‑speech gesture generation, combining 13 objective metrics and a novel Semantic Gesture Preservation (SGP) metric that measures how well ground‑truth semantic gestures are retained. A perceptual study with 101 participants rates generated gestures on five dimensions, revealing that SGP aligns with speech‑aware human judgments while other metrics do not. Composite metrics tailored to each dimension further improve alignment, especially for animation errors and content match, underscoring the need to validate objective metrics against human perception.
By Nick Milkin, Lanmiao Liu, Esam Ghaleb, Asli Ozyurek, Zerrin Yumak
arXiv:2609.00369v1 Announce Type: new
Abstract: Generating co-speech gestures that are temporally coherent, semantically aligned with speech, and grounded with surrounding objects remains challenging...
By Vida Adeli, Soroush Mehraban, Jacob Rommann, Harrison Sanborn, Cole Clifford, Babak Taati
Generating co-speech gestures that are temporally coherent, semantically aligned with speech, and grounded with surrounding objects remains challenging. Prior speech-driven gesture models emphasize au...
The paper introduces ELLSA 2, an end‑to‑end full‑duplex dialogue model that jointly optimizes speech interaction and streaming gesture‑expression generation. It uses an autoregressive, chunk‑wise synthesizer that combines pretrained acoustic features with distinct listening and speaking hidden states, enabling continuous non‑verbal generation even when the model is silent. The authors also present contextual semantic calls and template guidance to separate the decision to invoke a semantic gesture from its realization, and they establish a curated data and evaluation pipeline for dialog‑aware co‑speech gesture generation, showing that ELLSA 2 outperforms cascaded baselines and that semantic template guidance significantly improves gesture quality.
By Siyin Wang, Wenyi Yu, Terumi Chiba, Xianzhao Chen, Xiaohai Tian, Jun Zhang, Lu Lu, Chao Zhang
Pointing-based visual grounding requires models to precisely locate target objects by deciphering complex spatial relationships between the visual scene and pointing gestures. Traditional methods typically encode input images into static feature representations and perform reasoning primarily within the linguistic domain, often overlooking the rich perceptual cues and explicit spatial geometry inherent in images.
InteractGesture is a model‑agnostic, inference‑time method that enables fine‑grained spatial control of individual joints in continuous streaming co‑speech gesture generation. It guides diffusion sampler latent estimates through a differentiable RVQ‑VAE decoder, backpropagating spatial control gradients to adjust motion latents during sampling. To address chunk‑wise dependency issues in streaming generation, the method introduces Progressive Chunk Guidance, a chunk‑window strategy that keeps an active set of editable chunk latents with staggered delays, allowing spatial constraints to propagate gradients backward across chunk boundaries and reducing boundary inconsistencies.
By Ekkasit Pinyoanuntapong, Ajinkya Deogade, Paul Streli, Wenjing Zhang, Joanna Materzynska, Pu Wang, Vittorio Ferrari, Jie Shen
arXiv:2609.23797v1 Announce Type: cross
Abstract: Human motion is shaped by both external acoustic events and behavioral intent: spatial audio conveys environmental cues that elicit or guide a respon...
By Shuyang Xu, Zhiyang Dou, Yiduo Hao, Zekun Li, Liang Pan, Jingbo Wang, Cheng Lin, Yuan Liu, Wenping Wang, Mingmin Zhao, Taku Komura
arXiv:2606. 02962v1 Announce Type: cross Abstract: Egocentric Natural Language Query (NLQ) grounding asks a model to localize, in a long first-person video, the temporal interval that answers a free-form text query.
By Enmin Zhong, Carlos R. del-Blanco, Fernando Jaureguizar, Narciso Garc\'ia
The paper introduces a metric interaction framework for robotic manipulation that explicitly models object- and scene-level interactions in Cartesian space. It uses Interaction‑Centric Tokens (ICTs) to represent end‑effector trajectories relative to objects and a Metric Action Interaction Field (MAIF) to attend to scene point‑cloud features for geometry‑conditioned action corrections. Experiments show modest but consistent improvements across several benchmarks, including LIBERO, RoboTwin 2.0, and real‑world tasks.
By Lijie Wang, Zheng Lu, Yiming Wang, Heyang Yu, Kenghou Hoi, Bowen Hu, Di Cui, Tianyu Xin, Haoran Liao, Wanqi Zhong, Xingjie Fan, Yizhao Xu, Ziliang Wang, Fei Gao, Yiming Li
GestAdapt is a framework that generates co‑speech gestures conditioned on a specified wrist workspace, allowing humanoid robots to adapt their motions to environmental constraints such as walls. The system learns from six co‑speech corpora using a shared motion representation and can be retargeted to different robot embodiments. Experiments show that GestAdapt’s motions stay close to real‑motion distributions, achieve higher quality scores than a no‑workspace baseline, and outperform other methods in real‑robot evaluations on the Reachy2 humanoid.
By Bosong Ding, Xianglin Zhang, Miao Xin, Murat Kirtay, Giacomo Spigler
arXiv:2607. 16322v1 Announce Type: cross Abstract: Micro-gesture recognition demands the detection of fleeting, spatially localized movements that are frequently overwhelmed by dominant static appearances and background noise.
By Taorui Wang, Wei Xia, Hui Ma, Zijia Song, Jiayu Zhang, Zeheng Wang, Yong Xu, Zitong Yu
arXiv:2610.19723v1 Announce Type: new
Abstract: Interpreting an ambiguous utterance in real-world interactions often requires context beyond a single utterance. When the utterance refers to objects i...
By Shun Inadumi, Lee Sangmyeong, Koichiro Yoshino