The paper introduces Omni Demand Understanding (ODU), a benchmark designed to test whether multimodal models can infer a user's underlying demand from complex audio‑visual interactions. ODU requires models to detect the presence of a demand and infer intent using multimodal and conversational context, evaluated across single‑turn and multi‑turn scenarios. The authors built ODU‑Bench through a taxonomy‑guided approach, agentic video generation, and human‑recorded interactions, and found that even top models like Gemini 3.1 Pro recover only 44.7% of key information, with many models exhibiting high false‑trigger rates.
By Qi Chen, Yunfei Chu, Haolin He, Yifan Yang, Zihan Liu, Yuxuan Wang, Ziyang Ma, Ruiyang Xu, Meng Gao, Yinsong Yan, Ling Wang, Hui Wang, Wen Huang, Yiheng Chen, Guanrou Yang, Qiuqiang Kong, Jin Xu, Xie Chen
The article surveys multi‑turn conversational AI, highlighting its shift from isolated text prompts to sustained, multimodal interactions that involve clarifying goals, revising requests, and switching topics. It reviews literature across text‑only dialogue, AudioLLMs, multimodal and omni‑modal systems, and tool‑augmented agents, organizing findings around datasets, models, training, evaluation, and cross‑cutting challenges. The analysis reveals that while multimodal perception and action have progressed rapidly, systems still struggle with persistent memory, cross‑turn grounding, full‑duplex interaction, robust evaluation, and cultural alignment.
By Syeda Faiza Ahmed, Zien Sheikh Ali, Hunzalah Hassan Bhatti, Firoj Alam, Shammur Absar Chowdhury
arXiv:2603.16783v2 Announce Type: replace
Abstract: Robust voice agents require exposure to the full diversity of how people interact through speech. However, obtaining enough spoken interactions is...
By Jonggeun Lee, Junseong Pyo, Jeongmin Park, Yohan Jo
SimXAI is a lightweight, LLM‑centric conversational layer that replaces traditional intent recognition, dialogue management, and template‑based response modules in GUS‑style ConvXAI agents. By using a single tool‑calling LLM to interface with existing XAI resources, it provides a modular interface that improves function‑selection accuracy, faithfulness, and handling of context‑dependent requests. In a user study, substituting SimXAI for the conversational layer of the TalkToModel agent led to more accurate task completion in fewer turns, demonstrating enhanced conversational quality while preserving faithfulness.
By Van Bach Nguyen, Qianli Wang, Nils Feldhus, J\"org Schl\"otterer, Christin Seifert
arXiv:2610.17738v1 Announce Type: new
Abstract: End-to-end (E2E) full-duplex spoken language models (FD-SLMs) enable simutanous listening and speaking continously, requiring models to take appropriat...
By Jiaheng Dong, Shihong Tan, Gongping Huang, Jean Honorio, Abhirup Ghosh, Ting Dang
The article reports that people increasingly use multiple multimodal large language models (MLLMs) at the same time, choosing each model for its perceived strengths. This practice creates coordination challenges such as adapting prompts, calibrating trust, and managing separate conversation histories. A diary study and interviews with ten participants show that users build primary and secondary hierarchies among models, develop switching patterns based on task aggregation, and adjust effort, latency, and output credibility to manage multi‑MLLM workflows.
By Seunghwa Pyo, Donggun Lee, Jungwoo Rhee, Soobin Park, Youn-kyung Lim
arXiv:2609.08977v3 Announce Type: replace-cross
Abstract: In this work, we present Gander, a native multimodal duplex interaction model that builds on MiniCPM-o 4.5 and is further adapted for realtim...
By Orantqing, Shengpeng Ji, Junlong Tong, Jialong Zuo, Dongjie Fu, Di Cao, Yangzhuo Li, Shangda Wu, Franz, Evan, Theron Veyra, Changhao Pan, Jingyu Lu, Dongchao Yang, Zhifei Xie, Yang Tan, Xiaoyu Shen, Xiaoda Yang, Wenfu Wang, Teddy Sun, Steve Yves, Zhou Zhao
The paper introduces ReTurn, a benchmark of 7,000 multimodal tasks designed to evaluate how models selectively use conversational history. ReTurn includes pairs that test task-carrying history (Reconfirm/Reground) and evidence-carrying history (Retrieve/Rebind), each preserving the same target question, media, and answer. Experiments on 13 vision‑language and audio‑language models show a drop in open‑ended accuracy from 93.7% to 72.3% when history is involved, revealing that high question recall does not guarantee effective task application and that competing media can misdirect responses.
By Shuoyang Sun, Kerui Gu, Hao Fang, Shaoli Huang, Bin Chen
arXiv:2610.28405v1 Announce Type: new
Abstract: Recent advances in multimodal large language models (MLLMs) have substantially improved spoken interaction and video understanding. Existing online and...
By Yuhang Chen, Guangyue Liu, Shilong Liu
EgoArgus is a new, human‑annotated dataset that tests visual‑language models (VLMs) as situational assistants in five everyday dialogue‑video scenarios. It evaluates how well VLMs understand and decide when to intervene, especially when visual and textual cues are helpful, irrelevant, or conflicting. The study finds that current VLMs still struggle to reliably act as egocentric assistants and that existing modality‑bias mitigation methods offer limited improvement.
By Yu-Chien Tang, Yu-Hsiang Liu, An-Zi Yen
MIRAGE is a controlled study that examines how multimodal personal agents use historical evidence when conversation state changes. The study keeps evidence, questions, and scoring constant while varying only the conversation state, then checks if agents can determine answerability, recover the correct source, and answer from it. Results across seven multimodal backbones show distinct failure regimes before and after compaction, heavy reliance on context continuity by open-weight models, and mixed effects of retrieval pressure on source attribution.
By Yu Liu, Wenxiao Zhang, Cheng Hu, Cong Cao, Fangfang Yuan, Xinyu Wang, Jin B. Hong, Yanbing Liu
SpeechEQ is a new benchmark that evaluates the sociolinguistic reasoning of Speech‑Language Models (SLMs) by using a dataset of 2,265 dialogues covering 15 Emotional Quotient (EQ) subscales from EQ‑i 2.0 theory. The framework introduces a multi‑turn evaluation protocol and a Spoken EQ (SEQ) score inspired by human EQ assessments. Experiments reveal that current multimodal models suffer from a text‑reliant shortcut, an alignment‑induced safety trap, and contextual amnesia, limiting their emotional awareness in spoken interaction.
By Liang-Yuan Wu, Zih-Ching Chen, Tongshuang Wu, Chao-Han Huck Yang, Hua Shen