← Back to all news
arXiv Computer Vision By Ferdinand Paar, Lanmiao Liu, Asl{\i} \"Ozy\"urek, Serge Thill, Esam Ghaleb

DuoGesture: Motion-Grounded Semantic Conditioning and Biomechanical Beat Priors for Co-Speech Gesture Generation

Read the original on arXiv Computer Vision →

DuoGesture is a co‑speech gesture generation model that separates gesture synthesis into a semantic stream and a beat stream, coordinated by a Semantic Variational Information Bottleneck that decides when semantic gestures override rhythmic motion. The semantic stream uses Motion‑Grounded Semantic Conditioning, replacing word embeddings with motion‑language representations to provide motion‑aligned semantic priors for rare gesture triggers. The beat stream is regularised by an Inertial Beat Prior, an anthropometry‑weighted arm‑chain module that reduces jitter and improves rhythmic consistency. Experiments show DuoGesture outperforms strong baselines and ablations confirm the complementary roles of semantic grounding, stochastic stream selection, and biomechanical regularisation.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computer Vision.

Related stories

arXiv Computer Vision
Oct 9

Perceptually Grounded and Semantics-Aware Evaluation for Holistic Co-Speech Gesture Generation

The paper introduces a new benchmark for evaluating holistic, semantics-aware co‑speech gesture generation, combining 13 objective metrics and a novel Semantic Gesture Preservation (SGP) metric that measures how well ground‑truth semantic gestures are retained. A perceptual study with 101 participants rates generated gestures on five dimensions, revealing that SGP aligns with speech‑aware human judgments while other metrics do not. Composite metrics tailored to each dimension further improve alignment, especially for animation errors and content match, underscoring the need to validate objective metrics against human perception.

By Nick Milkin, Lanmiao Liu, Esam Ghaleb, Asli Ozyurek, Zerrin Yumak
llmsmultimodalbenchmarkssafety
More like this →
arXiv Computer Vision
Sep 1

RoboGesture: Real-Time Semantic-aligned Co-Speech Gestures Generation for Humanoid Interaction

arXiv:2608.28693v1 Announce Type: cross Abstract: Enabling humanoid robots to respond to human speech with synchronized and semantically meaningful gestures is fundamental to natural human-robot inte...

By Zifan Wang, Ziang Ren, Pengyang Shi, Zirui Wang, Chenghuai Lin, Tianze Wang, Zekun Qi, Liangliang Zhao, He Wang, Li Yi
llmsdiffusionroboticsbenchmarkssafety
More like this →
arXiv Computer Vision
Aug 27

InteractGesture: Progressive Chunk Guidance for Continuous Streaming Co-Speech Gesture Control

InteractGesture is a model‑agnostic, inference‑time method that enables fine‑grained spatial control of individual joints in continuous streaming co‑speech gesture generation. It guides diffusion sampler latent estimates through a differentiable RVQ‑VAE decoder, backpropagating spatial control gradients to adjust motion latents during sampling. To address chunk‑wise dependency issues in streaming generation, the method introduces Progressive Chunk Guidance, a chunk‑window strategy that keeps an active set of editable chunk latents with staggered delays, allowing spatial constraints to propagate gradients backward across chunk boundaries and reducing boundary inconsistencies.

By Ekkasit Pinyoanuntapong, Ajinkya Deogade, Paul Streli, Wenjing Zhang, Joanna Materzynska, Pu Wang, Vittorio Ferrari, Jie Shen
diffusion
More like this →