arXiv AI

When to Ask, When to Answer: Strategic Clarification for Ambiguous Reference Resolution in Dialogue

The paper investigates strategic clarification in dialogue, focusing on when large language models should ask clarification questions versus providing immediate answers to resolve ambiguous references. Using a controlled test environment derived from the SIMMC 2.1 dataset, the authors evaluate baseline LLM behavior and post‑training methods, noting a bias toward premature answering. They introduce BACE, a lightweight belief‑aware critic that tracks epistemic uncertainty, and demonstrate that it more consistently selects the optimal number of clarifying questions compared to methods like GRPO and ArCHer.

arXiv AI
Oct 23

A Small Model Asks Better: Fine-Tuning Query Clarification for Retrieval-Augmented Generation

The paper introduces ClariEdge, a dataset of about 3,900 examples that capture ambiguous or ill‑formed user queries in retrieval‑augmented generation contexts, including negative cases that do not need clarification. A lightweight LoRA fine‑tuned model achieves 96.6% classification accuracy, outperforming larger prompted baselines in both classification and clarification generation. The study highlights the importance of hard negatives for distinguishing truly ambiguous queries and demonstrates that the same model can be applied before and after retrieval in a RAG pipeline.

By Lihong He, Lucian Popa, Marina Danilevsky
arXiv Computation and Language
Sep 3

A Tri-Agent Framework for Evaluating and Aligning Question Clarification Capabilities of Large Language Models

The paper presents a tri‑agent framework for evaluating large language models’ question‑clarification abilities. It involves a Question Clarifying Agent that identifies ambiguities and asks follow‑up questions, a Respondent Agent that simulates human replies, and an Evaluator Agent that judges the dialogue using metrics such as ambiguity handling, question quality, dialogue efficiency, language appropriateness, and intent alignment. The authors illustrate the approach with synthetic supply‑chain data and discuss validating the evaluator against human judgments.

By Yikai Zhao, Saurabh Pandey, Pradeep Kumar Misra
arXiv Computation and Language
Nov 4

Don't Guess, Ask Wisely: Benchmarking and Training Proactive Elicitation in Open-Domain LLM Conversations

The paper introduces VagueBench, a benchmark that degrades seed queries into vague variants to evaluate large language models’ ability to ask follow‑up questions when user queries are incomplete. It proposes DialogGRPO, an offline reinforcement learning approach that models multi‑turn elicitation as a dialogue‑state directed acyclic graph, using dense turn‑level rewards to guide question selection. DialogGRPO achieves over 88% success in eliciting missing information on vague queries, improves answer coverage and recovery efficiency, and outperforms prompting and test‑time scaling baselines while generalizing zero‑shot to other dialogue datasets.

By Rongyu Zhang, Mutian Bao, Jinjian Zhang, Yuetian Du, He Xu, Wei Zhou, Xiaojun Zhu, Ming Kong, Jie Liu, Linjian Mo, Qiang Zhu