arXiv AI
Large-Scale User Behavior Analysis in Multimodal AI-Assisted Manual Task Execution
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv AI
Aug 19
Multi-turn Conversational AI from Text to Multimodal Interaction: Data, Models, Evaluation, and Open Challenges
The article surveys multi‑turn conversational AI, highlighting its shift from isolated text prompts to sustained, multimodal interactions that involve clarifying goals, revising requests, and switching topics. It reviews literature across text‑only dialogue, AudioLLMs, multimodal and omni‑modal systems, and tool‑augmented agents, organizing findings around datasets, models, training, evaluation, and cross‑cutting challenges. The analysis reveals that while multimodal perception and action have progressed rapidly, systems still struggle with persistent memory, cross‑turn grounding, full‑duplex interaction, robust evaluation, and cultural alignment.
arXiv Computation and Language
Sep 2
SpokenUS: A Spoken User Simulator for Task-Oriented Dialogue
arXiv:2603.16783v2 Announce Type: replace Abstract: Robust voice agents require exposure to the full diversity of how people interact through speech. However, obtaining enough spoken interactions is...
arXiv AI
4d ago
Multimodal Duplex Interaction Agent
arXiv:2609.08977v3 Announce Type: replace-cross Abstract: In this work, we present Gander, a native multimodal duplex interaction model that builds on MiniCPM-o 4.5 and is further adapted for realtim...
arXiv Computation and Language
Aug 27
EgoArgus: Benchmarking VLMs as Situational Assistants for Modality-Grounded User Supports
EgoArgus is a new, human‑annotated dataset that tests visual‑language models (VLMs) as situational assistants in five everyday dialogue‑video scenarios. It evaluates how well VLMs understand and decide when to intervene, especially when visual and textual cues are helpful, irrelevant, or conflicting. The study finds that current VLMs still struggle to reliably act as egocentric assistants and that existing modality‑bias mitigation methods offer limited improvement.
arXiv AI
2d ago
MIRAGE: How Conversation State Shapes Historical Evidence Use in Multimodal Personal Agents
MIRAGE is a controlled study that examines how multimodal personal agents use historical evidence when conversation state changes. The study keeps evidence, questions, and scoring constant while varying only the conversation state, then checks if agents can determine answerability, recover the correct source, and answer from it. Results across seven multimodal backbones show distinct failure regimes before and after compaction, heavy reliance on context continuity by open-weight models, and mixed effects of retrieval pressure on source attribution.
arXiv Machine Learning
Jun 19
Proactive Conversational Assistant for a Procedural Manual Task based on Audio and IMU
arXiv:2602. 15707v2 Announce Type: replace-cross Abstract: Real-time conversational assistants for procedural manual tasks often depend on video input, which can be computationally expensive and compromise user privacy.