arXiv AI September 10, 2026 By Hang Zhang, Chaokun Wang, Yuzhi Pan, Ziyao Zhong, Shuo Cao, Yue Xue, Zeyu Huang, Xingwei Zhou, Fang Niu, Bofan Xie, Guanchen Ge, Leqi Zheng, Ziyang Liu, Xiannian Cao, Pengcheng Ge

ProcArena: A Multi-Scenario Benchmark for LLMs on Direct and Interactive PL/SQL Development from Natural Language

Read the original on arXiv AI →

ProcArena is a new benchmark for evaluating large language models on natural‑language to PL/SQL translation tasks. It contains 3,998 executable tasks across 157 databases, covering nine development subscenarios in PostgreSQL and Oracle, and supports both direct generation and interactive multi‑turn scenarios. Experiments on seven models show that even the best performers achieve only about 62% accuracy in direct mode and 58% in interactive mode, highlighting the difficulty of realistic NL‑to‑PL/SQL development.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

arXiv AI

Jun 11

TAHOE: Text-to-SQL with Automated Hint Optimization from Experience

arXiv:2606. 12387v1 Announce Type: cross Abstract: Large Language Models (LLMs) have democratized database access through Text-to-SQL, but moving from prototypes to production remains difficult.

By Zhiyi Chen, Jie Song, Peng Li

llms agents fine-tuning data-engineering

arXiv Machine Learning

Sep 22

Dial: A Knowledge-Grounded Dialect-Specific NL2SQL System

The paper introduces Dial, a knowledge‑grounded framework for generating SQL queries that respect the specific dialect of heterogeneous database systems. Dial comprises a Dialect‑Aware Logical Query Planning module, a hierarchical intent‑aware knowledge base (HINT‑KB) containing canonical syntax, function, and constraint repositories, and an execution‑driven debugging loop that separates syntactic recovery from logic auditing. The authors also present DS‑NL2SQL, a benchmark of 2,218 dialect‑specific test cases across six major database systems, and report that Dial improves translation accuracy by 10.25% and dialect feature coverage by 15.77% over existing baselines.

By Xiang Zhang, Hongming Xu, Le Zhou, Wei Zhou, Xuanhe Zhou, Guoliang Li, Yuyu Luo, Changdong Liu, Guorun Chen, Jiang Liao, Fan Wu

rag fine-tuning benchmarks data-engineering

arXiv AI

Sep 10

SQLMorph: Query Mutation and Fine-Grained Metrics for Text-to-SQL Evaluation

arXiv:2609.08950v1 Announce Type: cross Abstract: Text-to-SQL systems translate natural language queries into executable SQL, democratizing access to structured data. Despite recent advances driven b...

By Mohammadhossein Malekpour, Mohamed Riahi, Maxime Lamothe, Amine Mhedhbi

llms benchmarks data-engineering

arXiv AI

Jun 16

Integrating Reasoning and Generalization in Text-to-SQL via Self-Enhanced Fine-Tuning

arXiv:2606. 15598v1 Announce Type: new Abstract: Text-to-SQL aims to translate natural language questions into executable SQL queries over structured databases, enabling non-expert users to access data intuitively.

By Feng Lyu, Jinfeng Cen, Sijing Duan, Hao Wu, Shucheng Li, Weixu Zhang, Haolun Wu

llms fine-tuning benchmarks data-engineering

arXiv AI

Sep 30

DatalogBench: Evaluating Large Language Models on Text-to-Datalog Synthesis

arXiv:2609.37233v1 Announce Type: cross Abstract: Datalog underpins reasoning tasks such as program analysis, but its programs are hard to write. Existing synthesizers automate this task but require...

By Yuan Li, Hanyun Jiang, Guowei Tian, Chengpeng Wang, Peisen Yao

llms agents benchmarks

arXiv AI

Aug 5

Evaluating LLMs in Database Scenarios: A Lifecycle Benchmark for Assessing Their Potential in Core Database Tasks

arXiv:2608. 03794v1 Announce Type: cross Abstract: Large Language Models (LLMs) are transforming database interaction paradigms, evolving from simple query translators to autonomous database administrators (DBAs).

By Shunfan Zheng, Dongsheng Shi, Yue Li, Xin Yi, Linlin Wang, Gerard de Melo

llms agents benchmarks data-engineering

One email a morning, machine-written

Related stories

TAHOE: Text-to-SQL with Automated Hint Optimization from Experience

Dial: A Knowledge-Grounded Dialect-Specific NL2SQL System

SQLMorph: Query Mutation and Fine-Grained Metrics for Text-to-SQL Evaluation

Integrating Reasoning and Generalization in Text-to-SQL via Self-Enhanced Fine-Tuning

DatalogBench: Evaluating Large Language Models on Text-to-Datalog Synthesis

Evaluating LLMs in Database Scenarios: A Lifecycle Benchmark for Assessing Their Potential in Core Database Tasks