arXiv AI By Oleg Grynets, Dmytro Kostetskyi, Vasyl Lyashkevych

Measuring What a Specification Determines: A Formal Semantic-Block Model and an Execution-Judged Benchmark

Read the original on arXiv AI →

arXiv:2608. 19475v1 Announce Type: cross Abstract: This work introduces a formal semantic-block model for specifications and an execution-judged benchmark for evaluating specification quality independently of model capability.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

Hugging Face Trending Papers
Aug 20

Specification-delta-driven data governance: an empirical study of the «spec-delta» as the unit of change in lakehouse data platforms

Spec Driven Development SDD has consolidated the idea that the specification rather than the code should be the primary artefact governing AI assisted work. Tools such as GitHub Spec Kit, and proposals such as Constitutional SDD, have formalised this principle in the software domain, while the executable data-contracts literature has extended it to schema and quality enforcement at run time.

arXiv AI
4d ago

ESQ-Bench: A Multi-Tier Enterprise Oracle Benchmark for Evaluating NL2SQL Dialect Generalization and Silent Semantic Divergence

ESQ‑Bench is an Oracle‑first NL2SQL benchmark that introduces systematic complexity tiers and silent‑divergence evaluation across three enterprise schema levels. It provides six populated schemas (465 tables, 164,682 rows) on Oracle, PostgreSQL, MySQL, and SQL Server, along with 550 gold‑validated question‑query pairs and a four‑metric evaluation harness. The benchmark reveals that state‑of‑the‑art models such as GPT‑4o and Claude Sonnet 4.6 experience significant drops in execution accuracy and high silent‑divergence rates as schema complexity increases, highlighting a gap between closed‑API models and open‑weight baselines on enterprise Oracle schemas.

By Sanjay Mishra, Divya Chukkapalli, Ganesh R. Naik
arXiv AI
2d ago

DRL: A Deterministic Relational Middleware Layer for Transaction-Safe Enterprise NL2SQL Under Schema-Graph Scaling

The paper introduces DRL, a deterministic relational middleware layer designed to enable transaction-safe natural‑language to SQL (NL2SQL) interfaces over large enterprise OLTP catalogs. DRL interposes between front‑ends and SQL back‑ends, employing dynamic context pruning, relational AST typing, and transactional safeguards (EXPLAIN gating and NULL guards) to keep context within LLM attention limits and detect silent divergence. Experiments on PostgreSQL and MySQL show significant context reductions (up to 92%) and high execution match rates (≈53%) for GPT‑4o, Claude Sonnet 4.5, and Gemini 2.5 Flash, while also revealing that evaluation code quality can materially affect reported performance gaps.

By Sanjay Mishra, Divya Chukkapalli, Ganesh R. Naik