← Back to all news
arXiv AI By Sanjay Mishra, Divya Chukkapalli, Ganesh R. Naik

ESQ-Bench: A Multi-Tier Enterprise Oracle Benchmark for Evaluating NL2SQL Dialect Generalization and Silent Semantic Divergence

Read the original on arXiv AI →

ESQ‑Bench is an Oracle‑first NL2SQL benchmark that introduces systematic complexity tiers and silent‑divergence evaluation across three enterprise schema levels. It provides six populated schemas (465 tables, 164,682 rows) on Oracle, PostgreSQL, MySQL, and SQL Server, along with 550 gold‑validated question‑query pairs and a four‑metric evaluation harness. The benchmark reveals that state‑of‑the‑art models such as GPT‑4o and Claude Sonnet 4.6 experience significant drops in execution accuracy and high silent‑divergence rates as schema complexity increases, highlighting a gap between closed‑API models and open‑weight baselines on enterprise Oracle schemas.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

Related stories

arXiv AI
2d ago

DRL: A Deterministic Relational Middleware Layer for Transaction-Safe Enterprise NL2SQL Under Schema-Graph Scaling

The paper introduces DRL, a deterministic relational middleware layer designed to enable transaction-safe natural‑language to SQL (NL2SQL) interfaces over large enterprise OLTP catalogs. DRL interposes between front‑ends and SQL back‑ends, employing dynamic context pruning, relational AST typing, and transactional safeguards (EXPLAIN gating and NULL guards) to keep context within LLM attention limits and detect silent divergence. Experiments on PostgreSQL and MySQL show significant context reductions (up to 92%) and high execution match rates (≈53%) for GPT‑4o, Claude Sonnet 4.5, and Gemini 2.5 Flash, while also revealing that evaluation code quality can materially affect reported performance gaps.

By Sanjay Mishra, Divya Chukkapalli, Ganesh R. Naik
llmsefficiencybenchmarkssafetydata-engineering
More like this →
arXiv AI
Jul 28

Schema-Aware Localisation (SAL): Live Schema Grounding and Hallucination Validation for Oracle NL2SQL

arXiv:2607. 22572v1 Announce Type: new Abstract: Large language models can generate fluent SQL from natural language, but on real enterprise Oracle databases they frequently fail at execution time: columns and aliases are hallucinated and dialect-specific syntax is missed, leading to ORA-00904 invalid-identifier errors.

By Sanjay Mishra, Divya Chukkapalli, Ganesh R. Naik
llmsagentssafetydata-engineering
More like this →
Hugging Face Trending Papers
Jun 29

How Far Do On-Prem Open LLMs Get on Text-to-SQL? A Cross-Family Size x Technique Frontier on BIRD

Organizations that cannot send data to a cloud API increasingly ask: how good is Text-to-SQL if the model must run on-premises on open weights, and which popular accuracy "recipes" are worth their compute? We answer with an honest, fully reproducible benchmark on the BIRD development split (n=1534, Execution Accuracy), evaluating three open model families across two generations -- Qwen2.

llmsragbenchmarksdata-engineering
More like this →