arXiv Computer Vision
Kandinsky S2V 1.0: Audio-Driven Avatar Video Generation
Read the original on arXiv Computer Vision →The Flow has not summarised this story yet — read it at arXiv Computer Vision.
Related stories
arXiv Computer Vision
Sep 23
Vorch-Human: Unified Multi-Task Human-Centric Generation via Long-Horizon Continuation
Vorch-Human is a unified framework for human‑centric audio‑visual generation that handles multiple tasks—animating a person from speech, jointly generating speech and video from a voice reference, and synthesizing a scene from paired appearance and voice references—using a single dual‑stream audio‑video diffusion transformer. The model incorporates clean condition‑audio and condition‑video tokens, per‑token task embeddings, temporal position types, condition masks, and a shared multimodal prompt encoder to express diverse inputs such as driving speech, timbre examples, first frames, and subject images. A two‑level data pipeline supplies the necessary supervision by extracting speech, appearance, and timbre annotations from clips and linking consistent identity and outfit references across videos, while a frozen‑prefix recurrence enables long‑form audio‑driven generation with reduced boundary discontinuity and identity drift.
arXiv Machine Learning
Sep 22
AVTR-1: Open Stack for Real-Time Interactive Avatars
arXiv:2609.22913v1 Announce Type: cross Abstract: Talking-head and dyadic models now achieve real-time inference, yet fast motion generation alone does not produce an interactive conversation. A live...
arXiv Computer Vision
Sep 7
Motion-Omni: End-to-End Joint Speech and Full-Body Motion for Spoken Dialogue
Motion-Omni is an end‑to‑end framework that jointly generates spoken dialogue and full‑body motion, producing speech, facial expressions, and hand, upper‑body, and lower‑body movements directly from the hidden states of a language model. The system requires joint training of the language model, speech generator, and motion generator to maintain audio‑motion alignment, and it is supervised using a scalable, model‑agnostic pipeline that pseudo‑labels 422,856 speech‑motion pairs. With a Qwen2.5‑7B‑Instruct backbone, Motion‑Omni‑Q7 achieves near‑cascade performance on motion metrics while being 5.4× faster, and it outperforms other non‑teacher cascades on beat correlation, diversity, and word error rate.
arXiv Computer Vision
Oct 1
MegaAvatar: Controllable Talking Avatar Generation
arXiv:2609.39273v1 Announce Type: new Abstract: This report presents \textbf{MegaAvatar}, a controllable talking avatar generation framework built on top of the Wan2.2-TI2V-5B model. Compared with pr...
arXiv Computer Vision
5d ago
Learning Visual Speech Representations via Cross-Modal Distillation and Joint Face-Lip Modeling
arXiv:2610.19559v1 Announce Type: new Abstract: Lipreading remains challenging due to the inherent ambiguity of visual speech cues and their sensitivity to head pose and video resolution. In this wor...
arXiv Computer Vision
1d ago
LiveSync: Iterative Self-Training for Lip-Sync Generation
arXiv:2610.23558v1 Announce Type: new Abstract: Audio-driven lip synchronization aims to match target speech through localized mouth and jaw articulation while preserving subject identity, head pose,...