arXiv AI

Conversational Task Disambiguation over Tabular Data: Leakage-Aware Formulation, Benchmark Suite, and Training

The paper introduces a leakage‑aware framework for conversational task disambiguation over tabular data, defining ambiguous verifiable tasks and separating the agent into an asking policy and a solution policy. It presents AmbiTab, a benchmark suite that unifies six ambiguous datasets under a common representation, enabling evaluation of clarification strategies and oracle leakage. The authors train an asking policy with reinforcement learning, achieving improved disambiguation metrics across all datasets and higher task success on five of them, while providing diagnostics to measure oracle leakage during training.

arXiv Computation and Language

Oct 8

ARCS: Towards Precise Text-to-SQL via Structured Disambiguation

The paper introduces ARCS, a new benchmark for text-to-SQL that focuses on naturally occurring ambiguities in user queries over real-world databases. It proposes a structured disambiguation approach, where ambiguity is resolved through explicit, constrained interactions instead of free-form dialogue. Experiments show that even advanced models struggle, with the best open-source model achieving only 27% execution accuracy.

By Yihao Hu, Yanlin Feng, Naoki Otani, Nikita Bhutani

llms benchmarks data-engineering

arXiv AI

Jul 31

Fewer Clarifications, Better Code: Benchmarking Cross-Session Personalized Ambiguity Adaptation in Coding Assistants

arXiv:2607. 26611v1 Announce Type: new Abstract: AI-assisted coding increasingly translates informal user intent into executable software, yet coding requests often contain ambiguities that recur in user-specific ways across tasks and sessions.

By Zijian Xu, Wenshuo Zhang, Zisen Qin, Rui Sheng, Yushi Sun, Huamin Qu, Chuhan Shi

llms benchmarks

arXiv AI

2d ago

A Small Model Asks Better: Fine-Tuning Query Clarification for Retrieval-Augmented Generation

The paper introduces ClariEdge, a dataset of about 3,900 examples that capture ambiguous or ill‑formed user queries in retrieval‑augmented generation contexts, including negative cases that do not need clarification. A lightweight LoRA fine‑tuned model achieves 96.6% classification accuracy, outperforming larger prompted baselines in both classification and clarification generation. The study highlights the importance of hard negatives for distinguishing truly ambiguous queries and demonstrates that the same model can be applied before and after retrieval in a RAG pipeline.

By Lihong He, Lucian Popa, Marina Danilevsky

rag fine-tuning

arXiv Computation and Language

3d ago

HiddenLens: A Multi-Layer LLM Framework for Detecting and Resolving Hidden Assumptions in Software Requirements

arXiv:2610.29996v1 Announce Type: cross Abstract: Natural-language software requirements often omit assumptions about performance, operating conditions, user behaviour, and domain constraints, affect...

By Tanu Singh, Aashna Suman, Vinod Patidar

llms fine-tuning

arXiv Computation and Language

Sep 15

Where to Look and What to Use: Retrieve-Localize-Generate for Long-Term Conversational Memory Question Answering

arXiv:2609.07093v2 Announce Type: replace Abstract: Retrieval-augmented generation (RAG) enables large language models (LLMs) to answer questions by accessing external knowledge and has been widely a...

By Yifan Wang, Xinkui Lin, Yongxiu Xu, Shen Gao, Ruochen Yang, Kun Huang, Yubin Wang, Jie Wu, Wei Liu, Jian Luan, Hongbo Xu, Shuo Shang

llms rag nlp benchmarks

arXiv Machine Learning

Jul 13

Forget Narrowly, Retain Broadly: Unlearning as an Asymmetric Generalization Problem

arXiv:2607. 09236v1 Announce Type: new Abstract: Machine unlearning in LLMs is the targeted removal of specific knowledge while preserving all other capabilities, critical for privacy and safety.

By Amit Peleg, Naman Deep Singh, Naama Pearl, Bibhabasu Mohapatra, Matthias Hein

llms benchmarks safety

arXiv AI

Aug 13

CLAIM: Leading Open-domain Active Clarification of Large Language Models with Uncertainty Measurement

arXiv:2608. 11631v1 Announce Type: new Abstract: In open-domain human-computer interaction scenarios, large language models (LLMs) frequently encounter user queries that are ambiguous or incomplete.

By Kuangzhao Yang, Ziliang Zhao, Zhicheng Dou

llms reinforcement-learning fine-tuning safety

arXiv AI

Sep 1

Ignorance or Incompetence? Constructing Knowledge-Gated, Verifiable Tasks for LLM Agents

arXiv:2608.30322v1 Announce Type: new Abstract: Professional agent tasks often depend on conventions that are absent from public corpora, yet benchmarks rarely control whether an agent has access to...

By Hanlin Tian, Minhao Li, Yu Mi, Sihan Zhu, Zhao Yang, Yuxiang Wang, Hongquan Zhu, Qiufei Hu

llms agents benchmarks

arXiv AI

Aug 7

Tytan: Interactive Neurosymbolic Construction of Analytic Semantic Schemas from Relational Data

arXiv:2608. 06331v1 Announce Type: cross Abstract: From natural-language query interfaces to automated report generation, data analysis tools need a description of the data: the real-world entities it contains, which columns function as measures or identifiers, and how tables connect into units of analysis.

By Donna Hooshmand, Shubham Shahi, Cameron Barrie, Abhratanu Dutta, Marko Sterbentz, Harper Pack, Kristian J. Hammond

llms benchmarks

arXiv AI

Jul 17

Automatically Evolving Prompt Guidelines for Task-Specific Optimization

arXiv:2607. 14105v1 Announce Type: cross Abstract: For Large Language Models to reliably answer user queries, users must clearly specify requirements, context, and constraints.

By Cedric Richter, Salah Ghamizi, Mike Papadakis

llms nlp benchmarks

arXiv AI

Jun 11

TAHOE: Text-to-SQL with Automated Hint Optimization from Experience

arXiv:2606. 12387v1 Announce Type: cross Abstract: Large Language Models (LLMs) have democratized database access through Text-to-SQL, but moving from prototypes to production remains difficult.

By Zhiyi Chen, Jie Song, Peng Li

llms agents fine-tuning data-engineering

arXiv Computation and Language

Sep 3

A Tri-Agent Framework for Evaluating and Aligning Question Clarification Capabilities of Large Language Models

The paper presents a tri‑agent framework for evaluating large language models’ question‑clarification abilities. It involves a Question Clarifying Agent that identifies ambiguities and asks follow‑up questions, a Respondent Agent that simulates human replies, and an Evaluator Agent that judges the dialogue using metrics such as ambiguity handling, question quality, dialogue efficiency, language appropriateness, and intent alignment. The authors illustrate the approach with synthetic supply‑chain data and discuss validating the evaluator against human judgments.

By Yikai Zhao, Saurabh Pandey, Pradeep Kumar Misra

llms agents benchmarks safety

Related stories