arXiv AI
DIVE: Benchmarking Full-Duplex Interaction with Visual Engagement in Continuous Streams
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv Computation and Language
Sep 21
Omni Demand Understanding: A Benchmark for Contextual User-Intent Inference in Multimodal Interaction
The paper introduces Omni Demand Understanding (ODU), a benchmark designed to test whether multimodal models can infer a user's underlying demand from complex audio‑visual interactions. ODU requires models to detect the presence of a demand and infer intent using multimodal and conversational context, evaluated across single‑turn and multi‑turn scenarios. The authors built ODU‑Bench through a taxonomy‑guided approach, agentic video generation, and human‑recorded interactions, and found that even top models like Gemini 3.1 Pro recover only 44.7% of key information, with many models exhibiting high false‑trigger rates.
arXiv AI
Sep 15
Multimodal Duplex Interaction Agent
arXiv:2609.08977v3 Announce Type: replace-cross Abstract: In this work, we present Gander, a native multimodal duplex interaction model that builds on MiniCPM-o 4.5 and is further adapted for realtim...
arXiv AI
Aug 28
Temporally-Grounded Language Generation: Towards Real-Time Vision-Language Models
The paper introduces Temporally-Grounded Language Generation (TGLG), a benchmark that tests vision‑language models on their ability to produce semantically accurate and temporally precise utterances in real‑time settings. It identifies perceptual updating and contingency awareness as key capabilities, curates datasets from sports broadcasting and egocentric interactions, and proposes the TRACE metric to jointly evaluate semantic similarity and temporal alignment. The authors also present VLM‑TSI, a model that interleaves visual and linguistic tokens in a time‑synchronized manner, achieving better performance than a strong baseline yet still showing modest overall results, underscoring the challenge of real‑time VLMs.
arXiv AI
Jun 2
Moment-Video: Diagnosing Temporal Fidelity of Video MLLMs on Momentary Visual Events
arXiv:2606. 02522v1 Announce Type: cross Abstract: Video multimodal large language models (MLLMs) have made rapid progress on general and long-form video understanding, yet their ability to preserve brief answer-critical visual evidence remains underexplored.
arXiv Computation and Language
Oct 9
When History Helps and Hurts: Selective History Use across Multimodal Turns
The paper introduces ReTurn, a benchmark of 7,000 multimodal tasks designed to evaluate how models selectively use conversational history. ReTurn includes pairs that test task-carrying history (Reconfirm/Reground) and evidence-carrying history (Retrieve/Rebind), each preserving the same target question, media, and answer. Experiments on 13 vision‑language and audio‑language models show a drop in open‑ended accuracy from 93.7% to 72.3% when history is involved, revealing that high question recall does not guarantee effective task application and that competing media can misdirect responses.
arXiv AI
Jun 2
Towards Understanding Modality Interaction in Multimodal Language Models via Partial Information Decomposition
arXiv:2606. 00959v1 Announce Type: new Abstract: Understanding modality interaction in multimodal large language models (MLLMs) is central to reliable deployment.