arXiv Machine Learning

Efficient Sequential Evaluation of Large Language Models

arXiv:2607. 17409v1 Announce Type: cross Abstract: We study the problem of sequentially evaluating a new large language model (LLM) on a fixed question set using historical performance data from prior LLMs.

arXiv AI

Jul 1

Test-Time Verification for Text-to-SQL via Outcome Reward Models

arXiv:2606. 30851v1 Announce Type: cross Abstract: Improving the reliability of large language models (LLMs) at inference time is a central challenge in structured reasoning tasks such as Text-to-SQL.

By Mattia Tritto, Giuseppe Farano, Dario Di Palma, Gaetano Rossiello, Fedelucio Narducci, Dharmashankar Subramanian, Tommaso Di Noia

llms benchmarks safety data-engineering

arXiv Computation and Language

Sep 1

QAQ: Bidirectional Semantic Coherence for Selecting High-Quality Synthetic Code Instructions

arXiv:2603.12165v3 Announce Type: replace Abstract: Synthetic data has become essential for training code generation models, yet it introduces significant noise and hallucinations that are difficult...

By Jiayin Lei, Ming Ma, Yunxi Duan, Chenxi Li, Tianming Yang

llms safety data-engineering

arXiv AI

Aug 5

Interpretable Adaptive Sampling for LLM Test-Time Scaling

arXiv:2608. 03961v1 Announce Type: new Abstract: Test-time scaling improves LLM reasoning by generating and aggregating multiple candidate answers, yet many pipelines use fixed per-query budgets that spend the same compute on easy and difficult prompts.

By Mobina Kashaniyan, Ali Jannesari

llms safety

arXiv AI

Jun 3

Test-Time Optimization of Physical Query Plans with LLMs

arXiv:2602. 10387v2 Announce Type: replace-cross Abstract: Traditional query optimization relies on cost-based optimizers that estimate execution cost (e.

By Mehmet Hamza Erol, Xiangpeng Hao, Federico Bianchi, Ciro Greco, Jacopo Tagliabue, James Zou

llms

arXiv AI

Jul 1

Optimal Self-Consistency for Efficient Reasoning with Large Language Models

arXiv:2511. 12309v2 Announce Type: replace-cross Abstract: Self-consistency (SC) is a widely used test-time inference technique for improving performance in chain-of-thought reasoning.

By Austin Feng, Marius Alonso, Ambroise Odonnat, Vasilii Feofanov, Ievgen Redko

llms benchmarks

arXiv AI

Sep 1

Automated Testing of LLM-Based Post Hoc Explainers Using Model Checking as an Oracle

arXiv:2608.30581v1 Announce Type: new Abstract: Large language models (LLMs) are used as post hoc explainers of sequential decision-making policies, producing natural-language explanations of why an...

By Dennis Gross, Helge Spieker

llms

arXiv AI

Aug 5

Test-Time Scaling in Reasoning LLMs: Inference Regimes, Evaluation, and Reproducibility

arXiv:2608. 04001v1 Announce Type: cross Abstract: Large language models can solve substantially harder reasoning problems with more inference-time compute.

By Mohsen Hariri, Weicong Chen, Nahal Shahini, Vikash Singh, Kai Ye, Amirhossein Samandar, Debargha Ganguly, Sreehari Sankar, Yanyan Zhang, Shouren Wang, Jerry Peng, Biyao Zhang, Michael Hinczewski, Vipin Chaudhary

llms benchmarks

arXiv AI

Sep 10

SQLMorph: Query Mutation and Fine-Grained Metrics for Text-to-SQL Evaluation

arXiv:2609.08950v1 Announce Type: cross Abstract: Text-to-SQL systems translate natural language queries into executable SQL, democratizing access to structured data. Despite recent advances driven b...

By Mohammadhossein Malekpour, Mohamed Riahi, Maxime Lamothe, Amine Mhedhbi

llms benchmarks data-engineering

arXiv Machine Learning

Jul 2

Clotho: Measuring Task-Specific Pre-Generation Test Adequacy for LLM Inputs

arXiv:2509. 17314v4 Announce Type: replace-cross Abstract: Software increasingly relies on the emergent capabilities of Large Language Models (LLMs), from natural language understanding to program analysis and generation.

By Juyeon Yoon, Somin Kim, Robert Feldt, Shin Yoo

llms benchmarks

arXiv Machine Learning

1d ago

Which LLM to pick? Online Active Model Selection for Large Language Models

The paper introduces ONLINE LLM PICKER, a framework for active model selection of large language models in streaming settings. It selects the most informative prompts for annotation within a limited budget, enabling the identification of the best or near‑best model among many candidates. Experiments on 10 datasets and over 130 language models show up to 71.67% savings in annotation cost and a reduction in regret by up to 2.51× when using the chosen model for sequential generation.

By Alessandro Turrin, Patrik Okanovic, Torsten Hoefler, Nezihe Merve G\"urel

llms benchmarks

arXiv AI

Sep 2

Replacing Training with Memory: Listwise Selection for Text-to-SQL

The paper proposes a fine‑tuning‑free listwise selector for Text‑to‑SQL systems that replaces traditional learning objectives with inference‑time strategies. It introduces reusable structured memories (MaP‑SQL) that encode mappings from natural language to schema elements, SQL operations, and expected outputs, and uses these memories to evaluate candidate queries. To reduce positional bias, the method aggregates rankings across multiple input permutations, optimizing inference cost through execution results and pointwise scoring. The approach achieves higher selection accuracy, fewer unnecessary comparisons, and outperforms the prior state‑of‑the‑art R^3‑SQL on the BIRD‑dev benchmark while using fewer tokens.

By Yeonseok Jeong, Soyoung Yoon, Seongjun Lee, Seung-won Hwang

llms fine-tuning benchmarks safety data-engineering

arXiv AI

Jun 2

Beyond the Frontier: Stochastic Backtracking for Efficient Test-Time Scaling

arXiv:2605. 25143v2 Announce Type: replace Abstract: Test-time scaling improves language model reasoning by spending additional compute to explore multiple solution trajectories.

By Dao Tran, Duc Anh Le, Ngoc Luu, Quan Pham, Tung Pham, Hung Bui

llms benchmarks

Related stories