arXiv Machine Learning

Query-efficient model evaluation using cached responses

arXiv:2605. 07096v2 Announce Type: replace Abstract: Evaluating a new model on an existing benchmark is often necessary to understand its behavior before deployment.

arXiv Machine Learning

Jul 21

Efficient Sequential Evaluation of Large Language Models

arXiv:2607. 17409v1 Announce Type: cross Abstract: We study the problem of sequentially evaluating a new large language model (LLM) on a fixed question set using historical performance data from prior LLMs.

By Chia-Yu Hsu, Shubhanshu Shekhar

llms

arXiv Machine Learning

Jun 19

Closing the Calibration Gap in Semantic Caching

arXiv:2606. 19719v1 Announce Type: cross Abstract: Semantic caching cuts LLM inference costs by serving a cached response to semantically similar queries.

By Aditeya Baral, Radoslav Ralev, Iliya Sotirov Zhechev, Srijith Rajamohan, Jen Agarwal

llms

arXiv AI

Jun 26

The Capability Frontier: Benchmarks Miss 82% of Model Performance

arXiv:2606. 26836v1 Announce Type: new Abstract: Existing benchmarks typically report accuracy for a single model on a single run.

By Bradley Fowler, Ryan Smith, Daniel Thi Graviet, William Myers, Joshua Greaves, Narmeen Fatimah Oozeer, Ant\'ia Garc\'ia, Philip Quirke, Amirali Abdullah, Fazl Barez, Shriyash Kaustubh Upadhyay

llms agents benchmarks

arXiv AI

Jun 3

Cost-Aware Query Routing in RAG: Empirical Analysis of Retrieval Depth Tradeoffs

arXiv:2606. 02581v1 Announce Type: cross Abstract: Retrieval-augmented generation (RAG) faces a fundamental three-way tension: deeper retrieval improves factual grounding but inflates token costs and end-to-end latency.

By Sanjay Mishra

llms rag benchmarks safety

arXiv AI

Jul 7

Risk-Constrained Freshness-Aware Semantic Caching for Open-Web Retrieval-Augmented LLMs

arXiv:2607. 04281v1 Announce Type: cross Abstract: Semantic caching reduces the latency and cost of retrieval-augmented generation (RAG) by serving cached answers to semantically similar queries, but most existing methods do not model the time-varying freshness of open-web evidence.

By Muhammad Mansoor, Tahir Ahmad, Yeo-Chan Yoon

llms rag benchmarks

arXiv AI

Aug 5

Getting the Parameters Right: A Difficulty-Graded Benchmark and Probe-Guided Training for LLM Tool Calls

arXiv:2608. 03071v1 Announce Type: new Abstract: Large language model agents derive much of their capability from tool use.

By Guoyao Yu, Xiaoqing Sun, Ziqi Huang, Shaojing Fan, Zhongyi Zhang, Xiaomeng Hu, Xiaobo Xue, Yangyang Shi, Xiong Xiao, Yang Song, Biao Lyu, Rong Wen, Xing Li, Qinming He, Shunming Zhu, Zhenguang Liu

llms rag agents fine-tuning benchmarks

arXiv AI

Jul 13

Attention to Detail: Evaluating Energy, Performance, and Accuracy Trade-offs Across vLLM Configurations

arXiv:2607. 09172v1 Announce Type: cross Abstract: Large Language Models are reshaping how software is developed and maintained.

By Nada Zine, Tristan Coignion, Vincenzo Stoico, Cl\'ement Quinton, Romain Rouvoy, Patricia Lago

llms

arXiv AI

Jun 3

Test-Time Optimization of Physical Query Plans with LLMs

arXiv:2602. 10387v2 Announce Type: replace-cross Abstract: Traditional query optimization relies on cost-based optimizers that estimate execution cost (e.

By Mehmet Hamza Erol, Xiangpeng Hao, Federico Bianchi, Ciro Greco, Jacopo Tagliabue, James Zou

llms

arXiv AI

Jul 24

MiniCache: Reusable Program Caching with Small Model Interfaces for Efficient LLM Inference

arXiv:2607. 20507v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly used for program-aided reasoning, agentic decision making, and structured task execution, but these applications often incur high inference cost.

By Jingquan Chen, Jinghua Piao, Jie Feng, Shaogang Hu, Yong Li

llms agents efficiency

arXiv Machine Learning

1d ago

Limits to scalable evaluation at the frontier: LLM as Judge won't beat twice the data

arXiv:2410. 13341v4 Announce Type: replace Abstract: High quality annotations are increasingly a bottleneck in the explosively growing machine learning ecosystem.

By Florian E. Dorner, Vivian Y. Nastl, Moritz Hardt

llms

arXiv AI

Jul 7

TabQueryBench: A Query-Centric Benchmark for Synthetic Tabular Data

arXiv:2607. 03926v1 Announce Type: cross Abstract: Synthetic tabular data support use cases like data sharing, model development under access restrictions, and rapid prototyping of analytical workflows.

By Jialin Zhang, Fenghao Dong, Yajie Zhou, Vyas Sekar, Shinan Liu

benchmarks safety data-engineering

arXiv AI

Jun 2

Query Circuits: Explaining How Language Models Answer User Prompts

arXiv:2509. 24808v2 Announce Type: replace Abstract: Explaining why a language model produces a particular output requires local, input-level explanations.

By Tung-Yu Wu, Fazl Barez

llms benchmarks

Related stories