arXiv AI October 16, 2026 By Morio Hamada, Jehyeok Yeon, Matteo Migliarini, Justin Olive, Magda Dubois

Evaluating Model Capabilities After Benchmark Saturation

The paper examines whether transcripts of model solutions can reveal capabilities after benchmark scores saturate. By simulating saturation with historical data, the authors test protocols that use only correctly solved transcripts to predict performance on unsaturated items. Pairwise judging of transcripts outperforms other methods, reducing prediction error by 5.3×, though its effectiveness can vary across distribution shifts.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

arXiv Machine Learning

Oct 8

Latent Performance Profiling of Large Language Models

arXiv:2605.30018v3 Announce Type: replace-cross Abstract: Large language models (LLMs) frequently achieve impressive scores on standardized benchmarks, yet accuracy alone offers a limited view of the...

By Tanmoy Chakraborty, Ayan Sengupta, Suparna Bhattacharya, Partha Pratim Chakrabarti, Amlan Chakrabarti, Supratik Chakraborty, Partha Pratim Das, Lipika Dey, Richa Singh, Mayank Vatsa

llms benchmarks safety

arXiv Computation and Language

Sep 4

SuperValid: Capability-Aligned OOD Validation for Generalizable Downstream Scaling

The paper introduces SuperValid, a framework that generates out-of-distribution, capability-aligned validation data by distilling core concepts from benchmarks and expanding them into diverse, knowledge-rich texts. By focusing on capability-level performance rather than benchmark-specific metrics, SuperValid’s loss correlates strongly and stably with downstream benchmark results across a wide range of models, scales, and training data distributions. This training‑free metric can be computed during training, enabling model selection, early stopping, and scaling decisions without the need for benchmark evaluation.

By Quanen Sun, Changxin Tian, Ke Shi, Cai Chen, Cunyin Peng, Jia Liu, Kunlong Chen, Zhiqiang Zhang, Jun Zhou

llms benchmarks

arXiv AI

Aug 3

Benchmarks Are Not Monolithic: Sample-Level Auditing and Orchestration for LLM Evaluation

arXiv:2607. 28801v1 Announce Type: cross Abstract: Benchmark datasets are central to evaluating Large Language Models (LLMs), yet they are typically conceived as monolithic tasks, obscuring substantial variation in the demands of individual samples.

By Philipp D. Siedler, Jordan Sassoon

llms agents benchmarks safety

arXiv AI

Jun 9

Prescriptive Scaling Reveals the Evolution of Language Model Capabilities

arXiv:2602. 15327v2 Announce Type: replace-cross Abstract: Machine learning model performance improvements tend to arise from competition and application.

By Hanlin Zhang, Jikai Jin, Vasilis Syrgkanis, Sham Kakade

llms benchmarks

arXiv AI

Sep 1

Measuring the Depth of LLM Unlearning via Activation Patching

arXiv:2605.24614v2 Announce Type: replace-cross Abstract: Large language model (LLM) unlearning has emerged as a crucial post-hoc mechanism for privacy protection and AI safety, yet auditing whether...

By Jaeung Lee, Dohyun Kim, Jaemin Jo

llms benchmarks safety

arXiv Machine Learning

Jul 24

DataPrep-Bench: Benchmarking LLMs as Training Data Preparators

arXiv:2607. 20465v1 Announce Type: new Abstract: The quality of training data fundamentally determines the capabilities of large language models (LLMs), yet no unified benchmark exists to measure how well LLMs, agents, and data-centric workflows actually prepare training data end to end.

By Hao Liang, Qifeng Cai, Yibo Lin, Jianzhuo Du, Qifeng Xia, Sizhe Qiu, Linzhuang Sun, Meiyi Qiang, Zhaoyang Han, Xiaochen Ma, Bohan Zeng, Ruichuan An, Conghui He, Wentao Zhang

llms agents fine-tuning benchmarks safety data-engineering

One email a morning, machine-written

Related stories

Latent Performance Profiling of Large Language Models

SuperValid: Capability-Aligned OOD Validation for Generalizable Downstream Scaling

Benchmarks Are Not Monolithic: Sample-Level Auditing and Orchestration for LLM Evaluation

Prescriptive Scaling Reveals the Evolution of Language Model Capabilities

Measuring the Depth of LLM Unlearning via Activation Patching

DataPrep-Bench: Benchmarking LLMs as Training Data Preparators