Natural-language interfaces to enterprise data must translate underspecified requests into governed, executable behavior while controlling invalid queries, policy failures, cost, and nondeterminism. SemPlan Benchmark evaluates this architectural design space with a deterministic synthetic bilingual benchmark containing 1,800 cases in English and Brazilian Portuguese; 1,200 cases form the frozen scientific evaluation subset.
arXiv:2608. 13612v1 Announce Type: new Abstract: Natural-language interfaces to enterprise data must translate underspecified requests into governed, executable behavior while controlling invalid queries, policy failures, cost, and nondeterminism.
arXiv:2608. 16663v1 Announce Type: cross Abstract: Direct text-to-SQL asks a language model to do two jobs: interpret the business question and construct the complete relational query.
arXiv:2609.22917v1 Announce Type: new Abstract: Non-technical stakeholders frequently cannot write the SQL needed to extract insights from operational databases. We built and evaluated a Text-to-SQL...
arXiv:2602. 16720v2 Announce Type: replace-cross Abstract: Text-to-SQL systems powered by Large Language Models have excelled on academic benchmarks but struggle in complex enterprise environments.
ESQ‑Bench is an Oracle‑first NL2SQL benchmark that introduces systematic complexity tiers and silent‑divergence evaluation across three enterprise schema levels. It provides six populated schemas (465 tables, 164,682 rows) on Oracle, PostgreSQL, MySQL, and SQL Server, along with 550 gold‑validated question‑query pairs and a four‑metric evaluation harness. The benchmark reveals that state‑of‑the‑art models such as GPT‑4o and Claude Sonnet 4.6 experience significant drops in execution accuracy and high silent‑divergence rates as schema complexity increases, highlighting a gap between closed‑API models and open‑weight baselines on enterprise Oracle schemas.
arXiv:2607. 22572v1 Announce Type: new Abstract: Large language models can generate fluent SQL from natural language, but on real enterprise Oracle databases they frequently fail at execution time: columns and aliases are hallucinated and dialect-specific syntax is missed, leading to ORA-00904 invalid-identifier errors.
text2ql is an open‑source Python framework that enables natural language querying of databases without relying on large language models at query time. It uses a language‑agnostic intermediate representation (QueryIR) and a pluggable renderer to support both SQL and GraphQL targets through a single seven‑stage detection pipeline. In deterministic mode, it achieves 100% execution accuracy with a median latency of 3.2 ms, while the LLM‑backed mode delivers 62‑70% exact match and 84‑91% execution accuracy on benchmark samples.
The paper introduces the DevRev NL2SQL benchmark, featuring 900 execution‑verified queries that test natural‑language‑to‑SQL systems on nested, graph‑like enterprise schemas, and proposes the Semantic Depth Score (SDS) as a rubric for analytical reasoning depth. It also presents a cost‑aware, single‑generation agentic architecture that includes schema selection, metadata retrieval, and error‑repair components tailored to these complex schemas. On the DevRev benchmark, the system achieves 91.7% answer correctness, outperforming the next‑best baseline by 54.6 percentage points, and remains competitive on the Spider 2.0 Snowflake dataset.
text2ql is an open‑source Python framework that enables natural language querying of databases without being limited to SQL, without requiring large language model inference at query time, and with a runtime confidence score for each generated query. It uses a language‑agnostic Intermediate Representation (QueryIR) and a pluggable renderer that supports both SQL and GraphQL through a single seven‑stage detection pipeline. In deterministic mode, it achieves 100% execution accuracy with a median latency of 3.2 ms, while the LLM‑backed mode reaches 62‑70% exact match and 84‑91% execution accuracy on benchmark samples.
arXiv:2608. 19475v1 Announce Type: cross Abstract: This work introduces a formal semantic-block model for specifications and an execution-judged benchmark for evaluating specification quality independently of model capability.
The paper introduces DRL, a deterministic relational middleware layer designed to enable transaction-safe natural‑language to SQL (NL2SQL) interfaces over large enterprise OLTP catalogs. DRL interposes between front‑ends and SQL back‑ends, employing dynamic context pruning, relational AST typing, and transactional safeguards (EXPLAIN gating and NULL guards) to keep context within LLM attention limits and detect silent divergence. Experiments on PostgreSQL and MySQL show significant context reductions (up to 92%) and high execution match rates (≈53%) for GPT‑4o, Claude Sonnet 4.5, and Gemini 2.5 Flash, while also revealing that evaluation code quality can materially affect reported performance gaps.
arXiv:2609.08950v1 Announce Type: cross Abstract: Text-to-SQL systems translate natural language queries into executable SQL, democratizing access to structured data. Despite recent advances driven b...