arXiv:2606. 29472v1 Announce Type: new Abstract: SWE-agent established the action interface as an underexplored design axis for software-engineering agents; we make the analogous case for the observation interface in computer-use (CU) agents.
arXiv:2606. 07547v1 Announce Type: cross Abstract: Speech-based large language models are typically constrained to spoken replies, which limits their user-facing outputs to what can be verbalized and suppresses text-native capabilities such as code generation, structured analysis, and multi-step reasoning in realtime interaction, for tasks that require persistent, structured, and inspectable intermediate outputs.
The technical report introduces Gander, an end‑to‑end model that integrates omni perception, real‑time interaction, and agentic capabilities into a single framework. Unlike traditional turn‑based systems, Gander continuously processes streaming inputs from video, speech, and text, enabling natural full‑duplex interaction in both everyday conversations and workflow‑oriented scenarios. Its architecture features a Cerebellum‑Brain collaboration—where the Cerebellum handles real‑time interaction and omni conversational tasks while the Brain manages complex reasoning—and a streaming Thinker‑Talker design that flattens inputs and outputs into an ordered token stream for low‑latency, continuous dialogue. Evaluations across conversational ability, omni understanding, interactive capability, and agentic intelligence show that Gander matches state‑of‑the‑art open‑source models in spoken dialogue while maintaining robust performance in noisy, multi‑party, and backchannel environments.
arXiv:2606. 05121v1 Announce Type: cross Abstract: Audio is an inherently interactive modality, yet today's Large Audio Language Models (LALMs) are offline, and streaming audio models each handle only a single task such as streaming ASR or voice chatting.
arXiv:2609.08977v3 Announce Type: replace-cross Abstract: In this work, we present Gander, a native multimodal duplex interaction model that builds on MiniCPM-o 4.5 and is further adapted for realtim...
arXiv:2608. 08469v1 Announce Type: new Abstract: Existing streaming multimodal models process observations incrementally but still follow a turn-based prefill-then-decode pattern, making them non-duplex: new observations cannot naturally enter an active generation stream.
DuplexSpeechBench-IFEval (DSB-IFEval) is a new benchmark that evaluates how full‑duplex voice agents follow implicit instructions during real‑time spoken interaction. It contains 1,038 test cases across eight assistant roles and tests five conditioning protocols, measuring floor management with an Instruction Adherence Score (IAS) and persona consistency with a Persona Adherence Score (PAS). Experiments on six speech systems reveal architecture‑dependent trade‑offs, showing that some models are more sensitive to explicit versus persona‑only instructions and that even when following conflicting directives, they struggle to override them under safety conflict.
arXiv:2608. 07631v1 Announce Type: cross Abstract: LLM-based full-duplex voice services allow users to speak while the assistant is responding.
SteerDuplex is a full‑duplex speech dialogue model that can be steered along attributes such as tone, persona, speaking rate, and voice style in response to user instructions. The authors introduce a taxonomy of text‑ and audio‑based steerability, identify gaps in existing models, and fine‑tune a Moshi‑based model with reinforcement learning to improve timing and response continuity. They also present SteerBench, a benchmark of 390 spoken prompts and 1,067 human‑authored rubrics, showing significant gains in audio‑steering pass rates and interruption handling compared to open baselines.
arXiv:2608. 10720v1 Announce Type: new Abstract: Omni-modal dialogue models can understand multimodal inputs and synthesize spoken replies, yet their responses remain visually disembodied.
arXiv:2606. 06991v1 Announce Type: cross Abstract: Online Video Large Language Models (Video-LLMs) have advanced toward seamless human-AI interaction through frame-by-frame processing and proactive responding.
Motion-Omni is an end‑to‑end framework that jointly generates spoken dialogue and full‑body motion, producing speech, facial expressions, and hand, upper‑body, and lower‑body movements directly from the hidden states of a language model. The system requires joint training of the language model, speech generator, and motion generator to maintain audio‑motion alignment, and it is supervised using a scalable, model‑agnostic pipeline that pseudo‑labels 422,856 speech‑motion pairs. With a Qwen2.5‑7B‑Instruct backbone, Motion‑Omni‑Q7 achieves near‑cascade performance on motion metrics while being 5.4× faster, and it outperforms other non‑teacher cascades on beat correlation, diversity, and word error rate.