The Flow has not summarised this story yet — read it at arXiv Computation and Language.
MIRAGE is a controlled study that examines how multimodal personal agents use historical evidence when conversation state changes. The study keeps evidence, questions, and scoring constant while varying only the conversation state, then checks if agents can determine answerability, recover the correct source, and answer from it. Results across seven multimodal backbones show distinct failure regimes before and after compaction, heavy reliance on context continuity by open-weight models, and mixed effects of retrieval pressure on source attribution.
The paper introduces ReTurn, a benchmark of 7,000 multimodal tasks designed to evaluate how models selectively use conversational history. ReTurn includes pairs that test task-carrying history (Reconfirm/Reground) and evidence-carrying history (Retrieve/Rebind), each preserving the same target question, media, and answer. Experiments on 13 vision‑language and audio‑language models show a drop in open‑ended accuracy from 93.7% to 72.3% when history is involved, revealing that high question recall does not guarantee effective task application and that competing media can misdirect responses.
UTILMEM is a new diagnostic benchmark that tests how conversational agents use long‑term memory, focusing on reasoning over dense histories, spotting implicitly relevant memories, synthesizing distributed evidence, and resisting interference from similar distractors. It contains 1,717 instances across five domains and evaluates a range of retrieval‑based and memory‑augmented systems. The study shows that strong performance on traditional factual recall does not guarantee effective memory utilization, highlighting a gap between retrieving information and integrating it into coherent, task‑oriented outputs.
Large language model agents are increasingly deployed in human-human interaction settings, such as meeting assistants and clinical documentation systems, where they must observe conversations and retain information for downstream queries. Unlike traditional human-assistant settings, these environments are inherently multimodal, involve complex discourse phenomena such as anaphora and deixis, and contain asynchronous or conflicting information from multiple participants.
arXiv:2608. 05381v1 Announce Type: new Abstract: Current Multimodal Large Language Models (MLLMs) can process diverse sensory inputs, yet their reasoning remains heavily biased toward a dominant modality, resulting in brittle cross-modal reasoning.
The paper compares two approaches to pragmatic intent recognition in human‑robot interaction: FISER, which uses structured symbolic reasoning, and CEMIR, which enriches textual prompts with synthetic narratives and images. Experiments across four large language models on the HandMeThat v2 benchmark show no statistically significant performance difference between the two methods, regardless of model size or ambiguity difficulty. The results indicate that when task constraints are already present in the text, multimodal enrichment offers no measurable advantage over structured reasoning.