DuoGesture is a co‑speech gesture generation model that separates gesture synthesis into a semantic stream and a beat stream, coordinated by a Semantic Variational Information Bottleneck that decides when semantic gestures override rhythmic motion. The semantic stream uses Motion‑Grounded Semantic Conditioning, replacing word embeddings with motion‑language representations to provide motion‑aligned semantic priors for rare gesture triggers. The beat stream is regularised by an Inertial Beat Prior, an anthropometry‑weighted arm‑chain module that reduces jitter and improves rhythmic consistency. Experiments show DuoGesture outperforms strong baselines and ablations confirm the complementary roles of semantic grounding, stochastic stream selection, and biomechanical regularisation.
By Ferdinand Paar, Lanmiao Liu, Asl{\i} \"Ozy\"urek, Serge Thill, Esam Ghaleb
arXiv:2608.28693v1 Announce Type: cross
Abstract: Enabling humanoid robots to respond to human speech with synchronized and semantically meaningful gestures is fundamental to natural human-robot inte...
By Zifan Wang, Ziang Ren, Pengyang Shi, Zirui Wang, Chenghuai Lin, Tianze Wang, Zekun Qi, Liangliang Zhao, He Wang, Li Yi
The paper introduces a real‑time framework for generating co‑speech gestures for digital humans, coupling a streaming speech response module with a causal multimodal autoregressive gesture generator that uses only current speech and motion history. It also presents an offline data synthesis pipeline for virtual companion dialogues and a self‑evolving training loop that incorporates user feedback to continually adapt the model. Experiments show the system achieves a better latency‑quality trade‑off, stronger speech‑motion synchronization, and higher user preference than existing baselines.
By Wentao Jiang, Youchen Xie, Haidi Fan, Yajing Chen, Xin Wang, Ye Shi, Jingya Wang
GestureFAR is a flow‑autoregressive framework that generates natural co‑speech gestures from streaming speech while preserving causality and continuous motion expressiveness. It autoregresses over continuous motion latents using a transformer for audio‑motion context and a flow‑matching head to sample the next latent. A head‑only flow distillation strategy further reduces latency by collapsing multi‑step flow sampling into a single network evaluation, enabling real‑time token‑causal generation with improved quality‑latency trade‑off on the BEAT2 benchmark.
By Pinxin Liu, Haiyang Liu, Jiahao Luo, Junhua Huang, Chunhao Zou, Luchuan Song
arXiv:2608. 04479v1 Announce Type: cross Abstract: Text-to-audio (TTA) generation has recently achieved remarkable progress in synthesizing realistic audio from natural language descriptions.
By Jinting Wang, Yuguang Yang, Shengyu Li, Yan Rong, Shan Yang, Xiaoda Yang, Li Liu
InteractGesture is a model‑agnostic, inference‑time method that enables fine‑grained spatial control of individual joints in continuous streaming co‑speech gesture generation. It guides diffusion sampler latent estimates through a differentiable RVQ‑VAE decoder, backpropagating spatial control gradients to adjust motion latents during sampling. To address chunk‑wise dependency issues in streaming generation, the method introduces Progressive Chunk Guidance, a chunk‑window strategy that keeps an active set of editable chunk latents with staggered delays, allowing spatial constraints to propagate gradients backward across chunk boundaries and reducing boundary inconsistencies.
By Ekkasit Pinyoanuntapong, Ajinkya Deogade, Paul Streli, Wenjing Zhang, Joanna Materzynska, Pu Wang, Vittorio Ferrari, Jie Shen
arXiv:2609.23797v1 Announce Type: cross
Abstract: Human motion is shaped by both external acoustic events and behavioral intent: spatial audio conveys environmental cues that elicit or guide a respon...
By Shuyang Xu, Zhiyang Dou, Yiduo Hao, Zekun Li, Liang Pan, Jingbo Wang, Cheng Lin, Yuan Liu, Wenping Wang, Mingmin Zhao, Taku Komura
arXiv:2606. 22726v2 Announce Type: replace Abstract: Choreographic motion generation poses unique challenges for AI, demanding precise semantic control over complex, temporally structured, and expressive full-body dynamics.
By Seong Jong Yoo, Siyuan Peng, Felix Gu, Stratis Aloimonos, Cornelia Ferm\"uller
arXiv:2609.00369v1 Announce Type: new
Abstract: Generating co-speech gestures that are temporally coherent, semantically aligned with speech, and grounded with surrounding objects remains challenging...
By Vida Adeli, Soroush Mehraban, Jacob Rommann, Harrison Sanborn, Cole Clifford, Babak Taati
DuoTok is a source‑aware dual‑track music tokenizer designed for vocal‑accompaniment generation. It first learns a semantic audio representation via self‑supervised pretraining, then refines source‑aware structure with feature‑replacement noise and multi‑task supervision (spectral reconstruction, source separation regularization, and an ASR head for lyric alignment). The encoder is frozen and hard‑routed codebooks for vocals and accompaniment are learned, while a diffusion decoder restores fine acoustic detail from the discrete tokens, achieving a favorable predictability‑fidelity trade‑off at ultra‑low bitrate across public benchmarks.
By Rui Lin, Zhiyue Wu, Jiahe Lei, Kangdi Wang, Weixiong Chen, Junyu Dai, Tao Jiang
Generating co-speech gestures that are temporally coherent, semantically aligned with speech, and grounded with surrounding objects remains challenging. Prior speech-driven gesture models emphasize au...
FireRedAudio is a 9‑billion‑parameter audio language model that separates continuous input representations for audio understanding and speech generation, enabling a single autoregressive LLM to perform tasks such as ASR, zero‑shot TTS, Instruct TTS, and semantic/acoustic speech editing. The model uses a dedicated Audio Encoder for recognition and a RedAE‑based pathway for generation, with the LLM directly generating text or conditioning a flow‑matching DiT to produce acoustic latents. Evaluations show competitive or leading performance in multilingual ASR, content‑accurate zero‑shot TTS, strong instruction following, and significant improvements in speech editing over prior work.
By Feiyu Shen, Fenglong Xie, Junjie Li, Kun Xie, Lei Xie, Xu Tang, Xuelong Geng, Yan Jia, Yao Hu, Yichen Han, Yichen Wu, Ziqi Dai, Junjie Chen, Kai Huang, Manzhen Wei, Yixuan Li