← Back to all news
arXiv Machine Learning By Jingcheng Ni, Zekun Li, Zhiyang Dou, Xiaoyan Cong, Ziyao Zeng, Hongyang Du, Randall Balestriero, Taku Komura, Srinath Sridhar

Sonic-AMG: Streaming Human Motion Generation from Spatial Audio

Read the original on arXiv Machine Learning →

Sonic-AMG is a novel framework that generates human motion in real time from spatial audio, the first to do so in a streaming manner. It encodes acoustic content and body-relative source geometry as separate tokens, then uses a causal transformer with a flow head to produce continuous motion latents that a causal decoder turns into motion segments. The system introduces rehearsal training and spatial feedback to mitigate training‑inference mismatches, achieving state‑of‑the‑art FID and R‑precision on the SAM benchmark while enabling coherent, real‑time reactions to sound events and source locations.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

Related stories

arXiv AI
Oct 7

WorldSonus: Bringing Sound to Worlds

arXiv:2610.08760v1 Announce Type: cross Abstract: Recent advances in world models have enabled increasingly realistic visual synthesis. However, these generated environments remain largely silent. Br...

By Pengjun Fang, Jingyi Fa, Kam Man Wu, Jiaming Wang, Haoyuan Huang, Yaguang Wu, Xiangjun Huang, Ziyang Ma, Weijia Chen, Hongyu Liu, Zeyue Tian, Qifeng Chen
diffusionroboticsbenchmarkssafety
More like this →
arXiv Computer Vision
Aug 27

InteractGesture: Progressive Chunk Guidance for Continuous Streaming Co-Speech Gesture Control

InteractGesture is a model‑agnostic, inference‑time method that enables fine‑grained spatial control of individual joints in continuous streaming co‑speech gesture generation. It guides diffusion sampler latent estimates through a differentiable RVQ‑VAE decoder, backpropagating spatial control gradients to adjust motion latents during sampling. To address chunk‑wise dependency issues in streaming generation, the method introduces Progressive Chunk Guidance, a chunk‑window strategy that keeps an active set of editable chunk latents with staggered delays, allowing spatial constraints to propagate gradients backward across chunk boundaries and reducing boundary inconsistencies.

By Ekkasit Pinyoanuntapong, Ajinkya Deogade, Paul Streli, Wenjing Zhang, Joanna Materzynska, Pu Wang, Vittorio Ferrari, Jie Shen
diffusion
More like this →