Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

15,794 stories · RSS feed

arXiv Machine Learning

Jul 30

Scores Are Not Decisions: Cost-Aware Stopping for Tool Acquisition in LLM Agents

arXiv:2607. 27083v1 Announce Type: new Abstract: As LLM agents increasingly depend on diverse external services such as search engines, databases, and connectors, agent harnesses face a fundamental tool-selection challenge: acquiring too few tools leaves the task under-informed, while too many adds cost, context load, and privacy exposure.

By Yicheng Feng, Yan Zhang, Yan Cheng, Wei Qi

llms agents fine-tuning benchmarks safety

arXiv Machine Learning

Jul 30

Inverse Learning of Latent Risk-Neutral Densities from Irregular Option Quotes

arXiv:2607. 27188v1 Announce Type: new Abstract: Accurate option prices do not imply accurate recovery of the latent risk-neutral density.

By Lennon J. Shikhman, Michael Galarnyk, Aadi Dash, Nicholas A. Welsh

llms benchmarks safety

arXiv Machine Learning

Jul 30

Archetypes or ability? Clustering for modelling student mathematical competence

arXiv:2607. 26063v1 Announce Type: cross Abstract: Personalised learning systems often assume that mathematical ability is combined of discrete abilities, acquired sequentially and dependent upon first acquiring foundational abilities, and students often report different strengths.

By Benjamin Mawdsley, Tom Quilter, Richard Turner, Sarah Jackson, Paul Edwards

benchmarks

arXiv Machine Learning

Jul 30

When benchmark inferences do not compose: Projectibility in AI evaluation

arXiv:2607. 26159v1 Announce Type: cross Abstract: An AI benchmark result rarely reaches a consequential claim in one step.

By Brett Reynolds

benchmarks

arXiv Machine Learning

Jul 30

The Art of Not Forgetting A Local Learning Architecture for Continual Learning

arXiv:2607. 26523v1 Announce Type: new Abstract: We introduce CMP (Cognitive Memory Primitive), a continual-learning architecture that repre?

By Ashmith Atmuri, Yashaswini Rao Bhogarajula

llms benchmarks

arXiv Machine Learning

Jul 30

ReCo: Reweighting GRPO Against Distributional Concentration

arXiv:2607. 26862v1 Announce Type: new Abstract: Group Relative Policy Optimization (GRPO) has become a standard reinforcement learning method for post-training language models.

By Junoh Park, Junseo Hwang, Wonguk Cho, Taesup Kim

llms reinforcement-learning benchmarks

arXiv Machine Learning

Jul 30

BayesAME: Bayesian Active Model Evaluation

arXiv:2607. 27023v1 Announce Type: new Abstract: Evaluating large generative models across benchmarks is time-consuming and computationally expensive.

By Paula Cordero Encinar, Taylan Cemgil, Arnaud Doucet, Virginia Aglietti, Silvia Chiappa

benchmarks

arXiv Machine Learning

Jul 30

HealthSLM-Bench: Benchmarking Small Language Models for Mobile and Wearable Healthcare Monitoring

arXiv:2509. 07260v5 Announce Type: replace-cross Abstract: Mobile and wearable healthcare monitoring play a vital role in facilitating timely interventions, managing chronic health conditions, and ultimately improving individuals' quality of life.

By Xin Wang, Ting Dang, Xinyu Zhang, Vassilis Kostakos, Michael J. Witbrock, Hong Jia

llms fine-tuning benchmarks safety

arXiv Machine Learning

Jul 30

One-Frame Calibration with Siamese Network in Facial Action Unit Recognition

arXiv:2409. 00240v2 Announce Type: replace-cross Abstract: Automatic facial action unit (AU) recognition is used widely in facial expression analysis.

By Shuangquan Feng, Virginia R. de Sa

fine-tuning benchmarks safety

arXiv Machine Learning

Jul 30

REAP: Automatic Curation of Coding Agent Benchmarks from Interactive Production Usage

arXiv:2604. 01527v4 Announce Type: replace-cross Abstract: Production deployment of AI coding agents requires fast, reproducible evaluation signals.

By Smriti Jha, Matteo Paltenghi, Chandra Maddila, Vijayaraghavan Murali, Shubham Ugare, Satish Chandra

llms agents benchmarks

arXiv Machine Learning

Jul 30

Budget-Aware LLM Discovery via Cost-Calibrated Frontier Utility

arXiv:2607. 26828v1 Announce Type: new Abstract: Large language models increasingly support scientific and algorithmic discovery through inference-time search over evaluated candidates.

By Yansen Zhang, Yilu Liu, Tianyu Liu, Jiamin Chen, Xiaokun Zhang, Kai Xie, Xue Liu, Chen Ma, Yiyan Qi

llms benchmarks

arXiv Machine Learning

Jul 30

Emergent Sparsity in Frozen Random CNN Feature Extractors for Deep Reinforcement Learning

arXiv:2607. 26059v1 Announce Type: new Abstract: We report a striking phenomenon: deep reinforcement learning agents trained with frozen, randomly initialized CNN feature extractors spontaneously develop extremely sparse fully-connected representations, without any sparsity-inducing objective.

By Scott M. Norton

agents reinforcement-learning efficiency benchmarks

arXiv Machine Learning

Jul 30

Structurally Separated Uncertainty in Supervised Latent Variable Models

arXiv:2602. 11219v2 Announce Type: replace Abstract: Predictive uncertainty is commonly decomposed into epistemic and aleatoric components, but standard decompositions often produce strongly correlated estimates because both quantities are derived from the same predictive distribution.

By Tanmoy Mukherjee, Marius Kloft, Pierre Marquis, Zied Bouraoui

benchmarks

arXiv Machine Learning

Jul 30

Compressed Video Aggregator: Content-driven Module for Efficient Micro-Video Recommendation

arXiv:2605. 08810v2 Announce Type: replace Abstract: We propose \textbf{Compressed Video Aggregator} (CVA), a lightweight micro-video recommendation module that decouples video information from preference learning.

By Yang Xiao, Huiyuan Chen, Kaiyuan Deng, Chao Jiang, Zinan Ling, Ruimeng Ye, Fei Wang, Xiaolong Ma, Bo Hui

rag diffusion benchmarks

arXiv Machine Learning

Jul 30

Entity Resolution in Practice: Lessons from a Self-Serve Pipeline

arXiv:2607. 26298v1 Announce Type: new Abstract: We built and evaluated a self-serve entity resolution (ER) system on six benchmarks spanning 864 to 5M records, and three lessons emerged that are absent from existing ER literature.

By Kaushik Pavani, Ganga Aluri, Pravin Jadhav, Neeraj Prasad, Kiran Sanka

benchmarks

arXiv Machine Learning

Jul 30

Knowledge before Reasoning: EC-Reason-Bench, a Training-Free Diagnostic Benchmark for LLM Enzyme Classification

arXiv:2607. 26397v1 Announce Type: cross Abstract: Enzyme function prediction is a hierarchical, knowledge-intensive form of protein function classification.

By Linyu Li, Zhi Jin, Yichi Zhang, Dongming Jin, Yuanpeng He, Huanyao Zhang, Xuan Zhang, Gadeng Luosang, Nyima Tashi

llms benchmarks safety

arXiv Machine Learning

Jul 30

PowerAtlas: Towards Electricity-Computing Co-Scheduling for Power Systems

arXiv:2607. 26710v1 Announce Type: new Abstract: The rapid growth of AI workloads is turning data centers into large-scale, volatile, yet spatiotemporally flexible grid loads, creating an urgent need for coordinated electricity-computing scheduling.

By Kaiwen Jiang, Siya Xu, Ziyue Zhu, Chao Yang, Anh Tuan Luu, Haoran Luo

llms agents benchmarks

arXiv Machine Learning

Jul 30

Meta-Learned Reward Shaping for Reinforcement Learning from Human Feedback

arXiv:2607. 26094v1 Announce Type: new Abstract: Reinforcement Learning from Human Feedback (RLHF) is the standard approach for aligning large language models with human preferences, but its quality is limited by static, task-agnostic reward models.

By Yunpeng Chu

llms reinforcement-learning benchmarks safety

arXiv Machine Learning

Jul 30

Flow Map Learning via Nongradient Vector Flow

arXiv:2607. 26398v1 Announce Type: new Abstract: Diffusion and flow-based models benefit from simple regression losses, but inference incurs significant overhead because sampling requires integration.

By Mark Goldstein, Anshuk Uppal, Raghav Singhal, Aahlad Puli, Rajesh Ranganath

diffusion benchmarks

arXiv Machine Learning

Jul 30

Enhancing Automated Machine Learning via Homogeneous Train-Test Splitting Methods

arXiv:2607. 26625v1 Announce Type: new Abstract: Accurate model evaluation in machine learning depends critically on how datasets are split into training and testing subsets.

By Yearn Tan Yin Tze, Charles Grellois

benchmarks

Get the daily briefing