The paper introduces a new benchmark for evaluating holistic, semantics-aware co‑speech gesture generation, combining 13 objective metrics and a novel Semantic Gesture Preservation (SGP) metric that measures how well ground‑truth semantic gestures are retained. A perceptual study with 101 participants rates generated gestures on five dimensions, revealing that SGP aligns with speech‑aware human judgments while other metrics do not. Composite metrics tailored to each dimension further improve alignment, especially for animation errors and content match, underscoring the need to validate objective metrics against human perception.
arXiv:2609.36685v1 Announce Type: new Abstract: Co-speech gesture generation aims to synthesize natural gestures that are both temporally synchronized with speech and semantically consistent with the...
DuoGesture is a co‑speech gesture generation model that separates gesture synthesis into a semantic stream and a beat stream, coordinated by a Semantic Variational Information Bottleneck that decides when semantic gestures override rhythmic motion. The semantic stream uses Motion‑Grounded Semantic Conditioning, replacing word embeddings with motion‑language representations to provide motion‑aligned semantic priors for rare gesture triggers. The beat stream is regularised by an Inertial Beat Prior, an anthropometry‑weighted arm‑chain module that reduces jitter and improves rhythmic consistency. Experiments show DuoGesture outperforms strong baselines and ablations confirm the complementary roles of semantic grounding, stochastic stream selection, and biomechanical regularisation.
arXiv:2608. 04479v1 Announce Type: cross Abstract: Text-to-audio (TTA) generation has recently achieved remarkable progress in synthesizing realistic audio from natural language descriptions.
The paper introduces a benchmark for spatially grounded gesture generation, featuring about 2,000 pointing-annotated clips from naturalistic VR dialogue with ground-truth 3D referents. It defines a task where systems must decide when, how, and where to point within conversational speech, and proposes a protocol that separates temporal alignment, spatial grounding, and perceived naturalness. A flow‑matching baseline (MM‑Conv‑Flow) and a retrieval‑based system are evaluated against captured human motion, revealing that geometric grounding can surpass human pointing quality without improving perceived naturalness, underscoring the need to measure referential gesture quality along distinct dimensions.
The paper introduces a real‑time framework for generating co‑speech gestures for digital humans, coupling a streaming speech response module with a causal multimodal autoregressive gesture generator that uses only current speech and motion history. It also presents an offline data synthesis pipeline for virtual companion dialogues and a self‑evolving training loop that incorporates user feedback to continually adapt the model. Experiments show the system achieves a better latency‑quality trade‑off, stronger speech‑motion synchronization, and higher user preference than existing baselines.
PAGE (Parts-Aware Guidance) is a diffusion-based framework that improves co-speech gesture portrait video generation by enhancing local spatial fidelity and global audio‑visual consistency. It introduces Parts‑Specific Reweighting during training to focus supervision on key regions like lips, eyes, and hands, and Consistency Classifier Guidance plus Differential Guidance during inference to correct audio‑visual mismatches and refine motion separately. The authors also present the Chinese Announcer Speech (CAS) dataset, a large‑scale Chinese benchmark that expands on the PATS dataset, and demonstrate that PAGE outperforms existing methods in perceptual quality, gesture diversity, and audio‑visual alignment.
arXiv:2608.28693v1 Announce Type: cross Abstract: Enabling humanoid robots to respond to human speech with synchronized and semantically meaningful gestures is fundamental to natural human-robot inte...
arXiv:2609.00369v1 Announce Type: new Abstract: Generating co-speech gestures that are temporally coherent, semantically aligned with speech, and grounded with surrounding objects remains challenging...
The paper introduces ELLSA 2, an end‑to‑end full‑duplex dialogue model that jointly optimizes speech interaction and streaming gesture‑expression generation. It uses an autoregressive, chunk‑wise synthesizer that combines pretrained acoustic features with distinct listening and speaking hidden states, enabling continuous non‑verbal generation even when the model is silent. The authors also present contextual semantic calls and template guidance to separate the decision to invoke a semantic gesture from its realization, and they establish a curated data and evaluation pipeline for dialog‑aware co‑speech gesture generation, showing that ELLSA 2 outperforms cascaded baselines and that semantic template guidance significantly improves gesture quality.
arXiv:2608. 20312v1 Announce Type: new Abstract: The capability to perceive and synthesize human-human interactions is fundamental to developing intelligent digital human systems.
arXiv:2608. 09930v1 Announce Type: cross Abstract: Automated Text-to-Speech (TTS) evaluation methods (Mean Opinion Score (MOS) predictors and Audio Large Language Models (Audio-LLM) judges) are expected to reflect human perception, yet it is unclear how well they capture the distinct aspects of speech that listeners actually perceive.
arXiv:2606. 30266v1 Announce Type: cross Abstract: Motion-language agents must possess the bidirectional capability to both understand human movement (motion-to-text, M2T) and generate it from natural language (text-to-motion, T2M).