PAGE (Parts-Aware Guidance) is a diffusion-based framework that improves co-speech gesture portrait video generation by enhancing local spatial fidelity and global audio‑visual consistency. It introduces Parts‑Specific Reweighting during training to focus supervision on key regions like lips, eyes, and hands, and Consistency Classifier Guidance plus Differential Guidance during inference to correct audio‑visual mismatches and refine motion separately. The authors also present the Chinese Announcer Speech (CAS) dataset, a large‑scale Chinese benchmark that expands on the PATS dataset, and demonstrate that PAGE outperforms existing methods in perceptual quality, gesture diversity, and audio‑visual alignment.
By Yabiao Wang, Jiafu Wu, Yicheng Xu, Teng Hu, Shengzhe Zhou, Yanxiao Sun, Jiangning Zhang, Ying Tai, Yong Liu, Ming-Hsuan Yang
The paper introduces a real‑time framework for generating co‑speech gestures for digital humans, coupling a streaming speech response module with a causal multimodal autoregressive gesture generator that uses only current speech and motion history. It also presents an offline data synthesis pipeline for virtual companion dialogues and a self‑evolving training loop that incorporates user feedback to continually adapt the model. Experiments show the system achieves a better latency‑quality trade‑off, stronger speech‑motion synchronization, and higher user preference than existing baselines.
By Wentao Jiang, Youchen Xie, Haidi Fan, Yajing Chen, Xin Wang, Ye Shi, Jingya Wang
arXiv:2609.00369v1 Announce Type: new
Abstract: Generating co-speech gestures that are temporally coherent, semantically aligned with speech, and grounded with surrounding objects remains challenging...
By Vida Adeli, Soroush Mehraban, Jacob Rommann, Harrison Sanborn, Cole Clifford, Babak Taati
GestureFAR is a flow‑autoregressive framework that generates natural co‑speech gestures from streaming speech while preserving causality and continuous motion expressiveness. It autoregresses over continuous motion latents using a transformer for audio‑motion context and a flow‑matching head to sample the next latent. A head‑only flow distillation strategy further reduces latency by collapsing multi‑step flow sampling into a single network evaluation, enabling real‑time token‑causal generation with improved quality‑latency trade‑off on the BEAT2 benchmark.
By Pinxin Liu, Haiyang Liu, Jiahao Luo, Junhua Huang, Chunhao Zou, Luchuan Song
Generating co-speech gestures that are temporally coherent, semantically aligned with speech, and grounded with surrounding objects remains challenging. Prior speech-driven gesture models emphasize au...
arXiv:2604.09057v3 Announce Type: replace
Abstract: Audio-video (AV) generation has recently made strong progress in perceptual quality and multimodal coherence, yet generating content with plausible...
By Junchao Liao, Zhenghao Zhang, Xiangyu Meng, Litao Li, Ziying Zhang, Siyu Zhu, Long Qin, Weizhi Wang