arXiv AI
Jun 17
Implicit vs. Explicit Prompting Strategies for LVLMs in Referential Communication
arXiv:2606. 17372v1 Announce Type: cross Abstract: Two recent studies (Jones et al.
LVLMs and Humans Ground Differently in Referential Communication
arXiv:2601. 19792v4 Announce Type: replace-cross Abstract: For generative AI agents to partner effectively with human users, the ability to accurately predict human intent is critical.
Related stories
arXiv Computation and Language
Sep 1
Which one is banana man? Evaluating vision-language models in multi-turn pragmatic interpretation
The study examines how vision‑language models handle multi‑turn pragmatic interpretation in iterated reference games, where participants repeatedly identify novel referents using language. Researchers compared human performance with that of several models, manipulating context by varying its amount, order, and relevance. While humans consistently performed well, the models could use prior context but struggled to build relevant context for effective interpretation, indicating missing core skills for efficient linguistic collaboration.
arXiv AI
Aug 26
When Seeing Is Not Enough: Benchmarking Interactive Visual Grounding in LVLMs
The paper introduces a controlled evaluation framework for interactive visual grounding in large vision-language models (LVLMs), examining how varying amounts of initial target information and dialogue affect performance. Experiments across four visual contexts and interaction protocols show that current LVLMs lag behind human baselines, especially when no initial description is given and information must be gathered through questions. The study also finds that LVLMs are poorly calibrated, often overestimating confidence, and that interactive grounding remains a significant challenge requiring visual matching, information seeking, and synthesis.
arXiv AI
Jun 9
Aligned but Not Partner-Specific: Distinguishing How Multimodal LLM Agents Succeed in Reference Games Without Human-Like Conventions
arXiv:2606. 08081v1 Announce Type: cross Abstract: Repeated reference games test whether interlocutors replace their initially long descriptions with shorter, partner-specific conventions grounded in shared interaction history.
arXiv Computation and Language
Sep 22
Pretrained Persona Mixture Models and Tandem Models for Human Simulation
arXiv:2609.22607v1 Announce Type: new Abstract: We argue here that the current dominant practice in LLM human simulation: prompting instruction-tuned assistant language models to role-play personas,...
arXiv AI
Aug 19
Multi-turn Conversational AI from Text to Multimodal Interaction: Data, Models, Evaluation, and Open Challenges
The article surveys multi‑turn conversational AI, highlighting its shift from isolated text prompts to sustained, multimodal interactions that involve clarifying goals, revising requests, and switching topics. It reviews literature across text‑only dialogue, AudioLLMs, multimodal and omni‑modal systems, and tool‑augmented agents, organizing findings around datasets, models, training, evaluation, and cross‑cutting challenges. The analysis reveals that while multimodal perception and action have progressed rapidly, systems still struggle with persistent memory, cross‑turn grounding, full‑duplex interaction, robust evaluation, and cultural alignment.
arXiv AI
2d ago
Aligned but Not Partner-Specific: How Multimodal LLM Agents Succeed in Reference Games Without Forming Conceptual Pacts
arXiv:2606.08081v2 Announce Type: replace-cross Abstract: Repeated reference games test whether interlocutors replace their initially long descriptions with shorter, partner-specific expressions grou...
arXiv Computation and Language
Sep 11
Inverse Turing Bench: Evaluating Language Models as Judges of Human vs. AI Dialogue
The paper introduces Inverse Turing Bench, a benchmark designed to assess how well language models can distinguish between human-only and human-AI dialogues in multi-turn text. It provides paired dialogue transcripts and evaluates models on correctly identifying the type of conversation. Preliminary results show GPTZero, Claude Opus-4.6, and GPT-5.5 achieving the highest accuracies of 89.41%, 77.92%, and 75.94% respectively, highlighting both the strengths and limitations of statistical versus semantic detection approaches.
arXiv AI
Sep 10
Human or Machine? A Preliminary Turing Test for Speech-to-Speech Interaction
The paper reports the first Turing test for speech‑to‑speech systems, gathering 2,968 human judgments on conversations between nine state‑of‑the‑art S2S systems and 28 humans. None of the evaluated systems passed the test, highlighting a clear gap in human‑likeness. The authors diagnose the failure with an 18‑dimension taxonomy, finding that paralinguistic cues, emotional expressivity, and conversational persona—not semantic understanding—are the main bottlenecks, and they propose an interpretable model for automatic human‑vs‑machine discrimination.
arXiv Computation and Language
Sep 1
DuplexGen: Adaptive Synthesis of Human-AI Turn-Taking Dialogues
arXiv:2607.26178v2 Announce Type: replace Abstract: Turn-taking is a central component of full-duplex interaction. Which turn-taking behaviors are appropriate varies with the scenario, yet current mo...
arXiv AI
Sep 25
Conversational DNA: A Visual Language and Interactive Atlas of Human and AI Dialogue
Conversational DNA is a visual language and interactive atlas designed to explore human and AI dialogue by mapping speaker strands, communicative bases, and directed pairings. It visualizes speaker switching, response distance, and contribution length through adjustable helix geometry, and covers 151,489 episodes across eight corpora totaling 1.57 million source records. The system improves precision@5 on Molweni motif queries from 58.8% to 77.2% and demonstrates how annotation coverage affects perceived collection differences.
arXiv AI
Jul 29
RefBench-PRO: Perceptual and Reasoning Oriented Benchmark for Referring Expression Comprehension
arXiv:2512. 06276v3 Announce Type: replace-cross Abstract: Referring Expression Comprehension (REC) is a vision-language task that localizes a specific image region based on a textual description.