ESQ‑Bench is an Oracle‑first NL2SQL benchmark that introduces systematic complexity tiers and silent‑divergence evaluation across three enterprise schema levels. It provides six populated schemas (465 tables, 164,682 rows) on Oracle, PostgreSQL, MySQL, and SQL Server, along with 550 gold‑validated question‑query pairs and a four‑metric evaluation harness. The benchmark reveals that state‑of‑the‑art models such as GPT‑4o and Claude Sonnet 4.6 experience significant drops in execution accuracy and high silent‑divergence rates as schema complexity increases, highlighting a gap between closed‑API models and open‑weight baselines on enterprise Oracle schemas.
By Sanjay Mishra, Divya Chukkapalli, Ganesh R. Naik
arXiv:2607. 22572v1 Announce Type: new Abstract: Large language models can generate fluent SQL from natural language, but on real enterprise Oracle databases they frequently fail at execution time: columns and aliases are hallucinated and dialect-specific syntax is missed, leading to ORA-00904 invalid-identifier errors.
By Sanjay Mishra, Divya Chukkapalli, Ganesh R. Naik
arXiv:2608. 16663v1 Announce Type: cross Abstract: Direct text-to-SQL asks a language model to do two jobs: interpret the business question and construct the complete relational query.
By Yi Ai
arXiv:2607. 17269v1 Announce Type: new Abstract: Large language models encode world models implicitly in neural weights, which exposes four structural risks in high-precision domains such as medicine and finance: hallucination, frozen knowledge, poor explainability, and poor modifiability.
By Zhanbo Li, Shifeng Wu, Xiangjin Meng, Wenjie Cai
The paper introduces the DevRev NL2SQL benchmark, featuring 900 execution‑verified queries that test natural‑language‑to‑SQL systems on nested, graph‑like enterprise schemas, and proposes the Semantic Depth Score (SDS) as a rubric for analytical reasoning depth. It also presents a cost‑aware, single‑generation agentic architecture that includes schema selection, metadata retrieval, and error‑repair components tailored to these complex schemas. On the DevRev benchmark, the system achieves 91.7% answer correctness, outperforming the next‑best baseline by 54.6 percentage points, and remains competitive on the Spider 2.0 Snowflake dataset.
By Yoga Sri Varshan Varadharajan, Ajay Yadav, Ritesh Goru, Prateek Chaudhury, Constantine Caramanis, Prateek Jain, Divyateja Pasupuleti, Sunil Kumar Pandey
arXiv:2608. 19475v1 Announce Type: cross Abstract: This work introduces a formal semantic-block model for specifications and an execution-judged benchmark for evaluating specification quality independently of model capability.
By Oleg Grynets, Dmytro Kostetskyi, Vasyl Lyashkevych