arXiv AI
Large-Scale User Behavior Analysis in Multimodal AI-Assisted Manual Task Execution
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv Computation and Language
Sep 21
Omni Demand Understanding: A Benchmark for Contextual User-Intent Inference in Multimodal Interaction
The paper introduces Omni Demand Understanding (ODU), a benchmark designed to test whether multimodal models can infer a user's underlying demand from complex audio‑visual interactions. ODU requires models to detect the presence of a demand and infer intent using multimodal and conversational context, evaluated across single‑turn and multi‑turn scenarios. The authors built ODU‑Bench through a taxonomy‑guided approach, agentic video generation, and human‑recorded interactions, and found that even top models like Gemini 3.1 Pro recover only 44.7% of key information, with many models exhibiting high false‑trigger rates.
arXiv AI
Aug 19
Multi-turn Conversational AI from Text to Multimodal Interaction: Data, Models, Evaluation, and Open Challenges
The article surveys multi‑turn conversational AI, highlighting its shift from isolated text prompts to sustained, multimodal interactions that involve clarifying goals, revising requests, and switching topics. It reviews literature across text‑only dialogue, AudioLLMs, multimodal and omni‑modal systems, and tool‑augmented agents, organizing findings around datasets, models, training, evaluation, and cross‑cutting challenges. The analysis reveals that while multimodal perception and action have progressed rapidly, systems still struggle with persistent memory, cross‑turn grounding, full‑duplex interaction, robust evaluation, and cultural alignment.
arXiv Computation and Language
Sep 2
SpokenUS: A Spoken User Simulator for Task-Oriented Dialogue
arXiv:2603.16783v2 Announce Type: replace Abstract: Robust voice agents require exposure to the full diversity of how people interact through speech. However, obtaining enough spoken interactions is...
arXiv Computation and Language
3d ago
SimXAI: A Simple Drop-in Layer Increases Interaction Quality and Understanding in Conversational XAI
SimXAI is a lightweight, LLM‑centric conversational layer that replaces traditional intent recognition, dialogue management, and template‑based response modules in GUS‑style ConvXAI agents. By using a single tool‑calling LLM to interface with existing XAI resources, it provides a modular interface that improves function‑selection accuracy, faithfulness, and handling of context‑dependent requests. In a user study, substituting SimXAI for the conversational layer of the TalkToModel agent led to more accurate task completion in fewer turns, demonstrating enhanced conversational quality while preserving faithfulness.
arXiv AI
Oct 15
DuplexAct: How Interactive Are Frontier Full-Duplex Spoken Language Models?
arXiv:2610.17738v1 Announce Type: new Abstract: End-to-end (E2E) full-duplex spoken language models (FD-SLMs) enable simutanous listening and speaking continously, requiring models to take appropriat...
arXiv AI
Oct 16
One Is Not Enough: How People Use Multiple AI Models in Everyday Life
The article reports that people increasingly use multiple multimodal large language models (MLLMs) at the same time, choosing each model for its perceived strengths. This practice creates coordination challenges such as adapting prompts, calibrating trust, and managing separate conversation histories. A diary study and interviews with ten participants show that users build primary and secondary hierarchies among models, develop switching patterns based on task aggregation, and adjust effort, latency, and output credibility to manage multi‑MLLM workflows.