arXiv:2609.36685v1 Announce Type: new
Abstract: Co-speech gesture generation aims to synthesize natural gestures that are both temporally synchronized with speech and semantically consistent with the...
By Zhirui Xing, Long Ye, Kaige Li, Ziyi Xu, Ming Meng
DuoGesture is a co‑speech gesture generation model that separates gesture synthesis into a semantic stream and a beat stream, coordinated by a Semantic Variational Information Bottleneck that decides when semantic gestures override rhythmic motion. The semantic stream uses Motion‑Grounded Semantic Conditioning, replacing word embeddings with motion‑language representations to provide motion‑aligned semantic priors for rare gesture triggers. The beat stream is regularised by an Inertial Beat Prior, an anthropometry‑weighted arm‑chain module that reduces jitter and improves rhythmic consistency. Experiments show DuoGesture outperforms strong baselines and ablations confirm the complementary roles of semantic grounding, stochastic stream selection, and biomechanical regularisation.
By Ferdinand Paar, Lanmiao Liu, Asl{\i} \"Ozy\"urek, Serge Thill, Esam Ghaleb
arXiv:2608. 04479v1 Announce Type: cross Abstract: Text-to-audio (TTA) generation has recently achieved remarkable progress in synthesizing realistic audio from natural language descriptions.
By Jinting Wang, Yuguang Yang, Shengyu Li, Yan Rong, Shan Yang, Xiaoda Yang, Li Liu
The paper introduces a benchmark for spatially grounded gesture generation, featuring about 2,000 pointing-annotated clips from naturalistic VR dialogue with ground-truth 3D referents. It defines a task where systems must decide when, how, and where to point within conversational speech, and proposes a protocol that separates temporal alignment, spatial grounding, and perceived naturalness. A flow‑matching baseline (MM‑Conv‑Flow) and a retrieval‑based system are evaluated against captured human motion, revealing that geometric grounding can surpass human pointing quality without improving perceived naturalness, underscoring the need to measure referential gesture quality along distinct dimensions.
By Anna Deichler, Rishabh Dabral, Fethiye Irmak Dogan, Anindita Ghosh, Jonas Beskow
The paper introduces a real‑time framework for generating co‑speech gestures for digital humans, coupling a streaming speech response module with a causal multimodal autoregressive gesture generator that uses only current speech and motion history. It also presents an offline data synthesis pipeline for virtual companion dialogues and a self‑evolving training loop that incorporates user feedback to continually adapt the model. Experiments show the system achieves a better latency‑quality trade‑off, stronger speech‑motion synchronization, and higher user preference than existing baselines.
By Wentao Jiang, Youchen Xie, Haidi Fan, Yajing Chen, Xin Wang, Ye Shi, Jingya Wang
PAGE (Parts-Aware Guidance) is a diffusion-based framework that improves co-speech gesture portrait video generation by enhancing local spatial fidelity and global audio‑visual consistency. It introduces Parts‑Specific Reweighting during training to focus supervision on key regions like lips, eyes, and hands, and Consistency Classifier Guidance plus Differential Guidance during inference to correct audio‑visual mismatches and refine motion separately. The authors also present the Chinese Announcer Speech (CAS) dataset, a large‑scale Chinese benchmark that expands on the PATS dataset, and demonstrate that PAGE outperforms existing methods in perceptual quality, gesture diversity, and audio‑visual alignment.
By Yabiao Wang, Jiafu Wu, Yicheng Xu, Teng Hu, Shengzhe Zhou, Yanxiao Sun, Jiangning Zhang, Ying Tai, Yong Liu, Ming-Hsuan Yang