The paper introduces a new benchmark for evaluating holistic, semantics-aware co‑speech gesture generation, combining 13 objective metrics and a novel Semantic Gesture Preservation (SGP) metric that measures how well ground‑truth semantic gestures are retained. A perceptual study with 101 participants rates generated gestures on five dimensions, revealing that SGP aligns with speech‑aware human judgments while other metrics do not. Composite metrics tailored to each dimension further improve alignment, especially for animation errors and content match, underscoring the need to validate objective metrics against human perception.
By Nick Milkin, Lanmiao Liu, Esam Ghaleb, Asli Ozyurek, Zerrin Yumak
arXiv:2609.00369v1 Announce Type: new
Abstract: Generating co-speech gestures that are temporally coherent, semantically aligned with speech, and grounded with surrounding objects remains challenging...
By Vida Adeli, Soroush Mehraban, Jacob Rommann, Harrison Sanborn, Cole Clifford, Babak Taati
Generating co-speech gestures that are temporally coherent, semantically aligned with speech, and grounded with surrounding objects remains challenging. Prior speech-driven gesture models emphasize au...
The paper introduces ELLSA 2, an end‑to‑end full‑duplex dialogue model that jointly optimizes speech interaction and streaming gesture‑expression generation. It uses an autoregressive, chunk‑wise synthesizer that combines pretrained acoustic features with distinct listening and speaking hidden states, enabling continuous non‑verbal generation even when the model is silent. The authors also present contextual semantic calls and template guidance to separate the decision to invoke a semantic gesture from its realization, and they establish a curated data and evaluation pipeline for dialog‑aware co‑speech gesture generation, showing that ELLSA 2 outperforms cascaded baselines and that semantic template guidance significantly improves gesture quality.
By Siyin Wang, Wenyi Yu, Terumi Chiba, Xianzhao Chen, Xiaohai Tian, Jun Zhang, Lu Lu, Chao Zhang
Pointing-based visual grounding requires models to precisely locate target objects by deciphering complex spatial relationships between the visual scene and pointing gestures. Traditional methods typically encode input images into static feature representations and perform reasoning primarily within the linguistic domain, often overlooking the rich perceptual cues and explicit spatial geometry inherent in images.
InteractGesture is a model‑agnostic, inference‑time method that enables fine‑grained spatial control of individual joints in continuous streaming co‑speech gesture generation. It guides diffusion sampler latent estimates through a differentiable RVQ‑VAE decoder, backpropagating spatial control gradients to adjust motion latents during sampling. To address chunk‑wise dependency issues in streaming generation, the method introduces Progressive Chunk Guidance, a chunk‑window strategy that keeps an active set of editable chunk latents with staggered delays, allowing spatial constraints to propagate gradients backward across chunk boundaries and reducing boundary inconsistencies.
By Ekkasit Pinyoanuntapong, Ajinkya Deogade, Paul Streli, Wenjing Zhang, Joanna Materzynska, Pu Wang, Vittorio Ferrari, Jie Shen