arXiv Computer Vision
JAVIS: Joint Audio-Visual Interaction and Streaming for Real-Time Conversational Agents
Read the original on arXiv Computer Vision →The Flow has not summarised this story yet — read it at arXiv Computer Vision.
Related stories
arXiv AI
Jun 30
FacePlex: Full-Duplex Joint Speech-Facial Motion Generation for Conversational Avatars
arXiv:2606. 30145v1 Announce Type: new Abstract: Natural face-to-face conversation requires real-time speech generation together with synchronized facial motion.
Hugging Face Trending Papers
Jun 29
FacePlex: Full-Duplex Joint Speech-Facial Motion Generation for Conversational Avatars
Natural face-to-face conversation requires real-time speech generation together with synchronized facial motion. Existing systems only partially address this problem: speech-only full-duplex models can generate speech in real time but do not produce facial motion, while audio-driven facial motion models animate a face from already available audio rather than jointly generating speech and motion online.
arXiv Computation and Language
Sep 15
Exploring Multimodal Turn-Taking Cues in Face-to-Face Conversation using Voice Activity Projection
arXiv:2609.14666v1 Announce Type: cross Abstract: Turn-taking is a fundamental component of spoken interaction, and while humans naturally rely on both verbal and non-verbal signals, dialogue systems...
arXiv Machine Learning
Sep 22
AVTR-1: Open Stack for Real-Time Interactive Avatars
arXiv:2609.22913v1 Announce Type: cross Abstract: Talking-head and dyadic models now achieve real-time inference, yet fast motion generation alone does not produce an interactive conversation. A live...
arXiv AI
Sep 10
Noise Adaptive Streaming Audio-Visual Speech Token Enhancement for Robust Full-Duplex Spoken Dialogue Models
The paper introduces AV-STE, a modular streaming audio‑visual front‑end that enhances corrupted semantic speech tokens using noisy audio and lip video before they reach a frozen speech LLM. By preserving the downstream dialogue model’s pretrained conversational abilities, AV‑STE improves response coherence from 1.42 to 1.91 in same‑dataset speaker interference scenarios while maintaining turn‑taking behavior. These gains also transfer to out‑of‑domain Seamless Interaction.
arXiv AI
Aug 12
Ex-Omni-2D: Expressive Omni-Modal Dialogue Models with Native Visual Presence
arXiv:2608. 10720v1 Announce Type: new Abstract: Omni-modal dialogue models can understand multimodal inputs and synthesize spoken replies, yet their responses remain visually disembodied.