arXiv AI
Multimodal Duplex Interaction Agent
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv AI
Sep 10
Omni Interaction Agent Technical Report
The technical report introduces Gander, an end‑to‑end model that integrates omni perception, real‑time interaction, and agentic capabilities into a single framework. Unlike traditional turn‑based systems, Gander continuously processes streaming inputs from video, speech, and text, enabling natural full‑duplex interaction in both everyday conversations and workflow‑oriented scenarios. Its architecture features a Cerebellum‑Brain collaboration—where the Cerebellum handles real‑time interaction and omni conversational tasks while the Brain manages complex reasoning—and a streaming Thinker‑Talker design that flattens inputs and outputs into an ordered token stream for low‑latency, continuous dialogue. Evaluations across conversational ability, omni understanding, interactive capability, and agentic intelligence show that Gander matches state‑of‑the‑art open‑source models in spoken dialogue while maintaining robust performance in noisy, multi‑party, and backchannel environments.
arXiv AI
Aug 19
Multi-turn Conversational AI from Text to Multimodal Interaction: Data, Models, Evaluation, and Open Challenges
The article surveys multi‑turn conversational AI, highlighting its shift from isolated text prompts to sustained, multimodal interactions that involve clarifying goals, revising requests, and switching topics. It reviews literature across text‑only dialogue, AudioLLMs, multimodal and omni‑modal systems, and tool‑augmented agents, organizing findings around datasets, models, training, evaluation, and cross‑cutting challenges. The analysis reveals that while multimodal perception and action have progressed rapidly, systems still struggle with persistent memory, cross‑turn grounding, full‑duplex interaction, robust evaluation, and cultural alignment.
arXiv AI
Oct 9
DuplexAgent-RSI: Recursive Harness Improvement for Full-Duplex Voice Agent Collaboration
DuplexAgent-RSI is a full‑duplex voice‑agent collaboration system that coordinates continuous listening and speaking with asynchronous delegation to reasoning and coding agents. Its harness is modular, consisting of six editable components, and a closed‑loop improvement process (Duplex‑Harness‑RSI) revises these modules based on interaction traces and failure diagnostics. Experiments show that this approach yields stronger spoken‑knowledge and executable‑tool performance than comparable delegated systems while preserving robust interruption response.
arXiv AI
Oct 21
DIVE: Benchmarking Full-Duplex Interaction with Visual Engagement in Continuous Streams
arXiv:2610.28405v1 Announce Type: new Abstract: Recent advances in multimodal large language models (MLLMs) have substantially improved spoken interaction and video understanding. Existing online and...
arXiv AI
Jul 21
Just A Rather Very Intelligent Spoken Agent
arXiv:2607. 16610v1 Announce Type: new Abstract: Long-horizon AI agents are becoming increasingly capable, yet their interaction with users remains surprisingly thin.
Hugging Face Trending Papers
Sep 12
Realtime-Venus: A full-duplex interaction system with asynchronous delegation
Natural interaction in digital and physical environments requires continuous perception and timely responses. Spoken dialogue relies on acoustic and linguistic cues, while video interaction also requi...