arXiv AI
XYBench: Can LLMs Respond Pragmatically to Queries with Misconceptions?
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv Machine Learning
Aug 31
RealSWE: A Compositional Evaluation of Coding Agents under Realistic User Requests
arXiv:2608.27831v1 Announce Type: cross Abstract: Coding agents are now commonly evaluated on the SWE-bench family of benchmarks, whose tasks are built from curated GitHub issues--long, structured, a...
arXiv AI
Jul 9
MIRA-Math: A Benchmark for Minimal Information Requesting and Mathematical Reasoning
arXiv:2607. 07391v1 Announce Type: new Abstract: Mathematical reasoning benchmarks typically provide all facts needed to solve each problem, while interactive benchmarks often mix reasoning with tools, retrieval, and long-horizon dialogue.
arXiv Computation and Language
Oct 16
PremiseSQL: Evaluating Premise Critique and Abstention in Text-to-SQL Systems
PremiseSQL introduces a benchmark of 1,800 flawed natural language to SQL (NL2SQL) queries, grouped into four categories of premise failures, alongside their original valid counterparts. The study evaluates large language models’ ability to critique flawed premises and abstain from generating misleading SQL, using an evaluation framework across three prompting settings. Results show that current models tend to trust premises and rarely critique them, while stronger critique-oriented prompting improves detection but also raises false invalid judgments and reduces SQL generation for valid queries.
arXiv Computation and Language
Oct 15
Measuring and Reducing Unusable Replies of LLM Agents
arXiv:2610.19659v1 Announce Type: new Abstract: People delegate multi-step work to LLM agents and read only the reply that the agent hands back. Current evaluation scores task completion and the fait...
arXiv AI
Jul 31
Ask don't tell: Reducing sycophancy in large language models
arXiv:2602. 23971v4 Announce Type: replace-cross Abstract: Sycophancy, the tendency of large language models to favour user-affirming responses over critical engagement, has been identified as an alignment failure, particularly in high-stakes advisory and social contexts.
Hugging Face Trending Papers
Aug 14
Never the Number: Structural Abstention for AI Systems Whose Answers Are Consumed as Fact
Large language models have made natural language interfaces to databases (NLIDB) newly credible, but LLM text-to-SQL systems fail in a way that matters for deployment: a hallucinated column or a mis-a...