Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

15,528 stories · RSS feed

arXiv AI

Jul 31

Adversarial Pragmatics for AI Safety Evaluation: A Diagnostic Framework and Seed Benchmark for Language-Mediated Control

arXiv:2607. 01153v3 Announce Type: replace-cross Abstract: Safety evaluations for language models increasingly depend on judgments about ambiguous natural-language behaviour: whether a model followed an instruction, refused appropriately, complied with a policy, or misreported progress in an agentic task.

By Brett Reynolds

llms agents benchmarks safety

arXiv Machine Learning

Jul 31

PlatformBid: An Auto-Bidding Benchmark from a Unified Advertising Platform's Perspective

arXiv:2607. 27265v1 Announce Type: new Abstract: Real-time bidding is central to computational advertising, comprising three elements: Supply Side Platform (SSP) selling ad impressions, Demand Side Platform (DSP) bidding for advertisers, and Ad Exchange conducting auctions between them.

By Shengtian Yang, Yewen Li, Peng Jiang, Zhiyi Lyu, Bo An, Peng Jiang, Qingpeng Cai, Lei Feng

reinforcement-learning benchmarks

arXiv Machine Learning

Jul 31

DoTime: A Synthetic Benchmark Generator for Interventional and Counterfactual Time Series

arXiv:2607. 27263v1 Announce Type: new Abstract: Most benchmarks for causal inference over time series are observational, small, or domain-specific, leaving interventional and counterfactual estimation under-served exactly where it matters most, such as in healthcare, policy evaluation, and climate science.

By Dennis Thumm, Billy Tim Anthony, Ying Chen

benchmarks

arXiv Machine Learning

Jul 31

EvoCause: LLM-Guided Evolution of Causal Graphs for Root Cause Analysis

arXiv:2607. 27290v1 Announce Type: new Abstract: Modern telecommunication, cloud, and microservice systems emit correlated alarm cascades when components fail.

By Lei Zan, Keli Zhang, Shifeng Xie, Jiale Zheng, Zehao Xiao, Zhiwei Dong, Ke Zhang, Ruichu Cai, Malik Tiomoko, Lujia Pan

llms benchmarks safety

arXiv Machine Learning

Jul 31

TIER-MoE: Trust-Informed Expert Routing via Conditional Modality Risk for Multimodal Fusion in Biomedical Classification

arXiv:2607. 27289v1 Announce Type: new Abstract: The promise of multimodal fusion lies in combining complementary sources of evidence, yet more evidence does not always yield a better prediction.

By Yu Chang, Anzhe Cheng, Chenwei Wu, Zhuoran Wang, Jiahao Chen, Tamoghna Chattopadhyay, Sophia I. Thomopoulos, Paul M. Thompson, Liyue Shen, Paul Bogdan

multimodal benchmarks

arXiv Machine Learning

Jul 31

Expanding Data-Agnostic Pivotal Instances Selection Models with Proximity Trees and Ensemble Learning

arXiv:2607. 27522v1 Announce Type: new Abstract: As decision-making processes grow more complex, machine learning tools have become essential for tackling business and societal challenges.

By Alessio Cascione, Mattia Setzu, Cristiano Landi, Paolo Maria Mancarella, Riccardo Guidotti

benchmarks

arXiv Machine Learning

Jul 31

Latent-Kernel Discrete Flow Maps for Few-Step Generation

arXiv:2607. 27529v1 Announce Type: new Abstract: Discrete diffusion and flow-matching models denoise a sequence over many steps, but to keep each step cheap, they factorize the transition across positions and decide every token independently.

By Mansoor Ahmed, Yue-Tsz Fan, Hemanth Venkateswara, Murray Patterson

llms diffusion benchmarks

arXiv Machine Learning

Jul 31

A Montage-Agnostic Encoder for Calibration-Light Cross-User Gesture Recognition from Surface Electromyography

arXiv:2607. 27565v1 Announce Type: new Abstract: Pattern-recognition control promises a myoelectric prosthesis that responds to many intended gestures rather than one or two, but the promise has stayed in the laboratory.

By Jethro Odeyemi, W. J. Zhang

benchmarks

arXiv Machine Learning

Jul 31

Back from the Future: Key-Value Cache Management by Counter-Causal Surprise

arXiv:2607. 27600v1 Announce Type: new Abstract: Key-value (KV) cache management through compression and eviction strategies has emerged as an important research direction in recent years.

By Stephen Gould, Anton van den Hengel

llms efficiency multimodal benchmarks

arXiv Machine Learning

Jul 31

Kalman Meets Curriculum: Efficient Dynamic Prompt Selection for Adaptive RL Finetuning

arXiv:2607. 27610v1 Announce Type: new Abstract: Reinforcement learning (RL) finetuning significantly enhances the reasoning capabilities of large language models (LLMs), yet its effectiveness critically depends on selecting prompts of appropriate difficulty for the current policy.

By Haodong Zhu, Yangyang Ren, Yanjing Li, Sheng Xu, Haiguang Liu, Linlin Yang, Baochang Zhang

llms reinforcement-learning fine-tuning benchmarks

arXiv Machine Learning

Jul 31

Tight Sample Complexity for Low-Rank Adaptation: Matching Bounds and Rank Selection

arXiv:2607. 27680v1 Announce Type: new Abstract: Low-Rank Adaptation (LoRA) has become the standard mechanism for fine-tuning large pretrained models, yet its statistical properties remain only partially understood.

By Arunan J

fine-tuning benchmarks

arXiv Machine Learning

Jul 31

Beyond Binary Rewards: A Comparative Study of Reward Design for Reinforcement Unlearning

arXiv:2607. 27968v1 Announce Type: new Abstract: Machine unlearning seeks to selectively remove specific knowledge from trained language models without full retraining, a growing necessity under privacy regulations such as GDPR and the EU AI Act.

By Efstratios Zaradoukas, Davide Gabrielli, Bardh Prenkaj, Gjergji Kasneci

llms reinforcement-learning efficiency benchmarks safety

arXiv Machine Learning

Jul 31

Contrastive Reinforced Policy Optimization via Privileged Self-Distillation

arXiv:2607. 28026v1 Announce Type: new Abstract: Recent advances in post-training Large Language Models (LLMs) increasingly rely on Reinforcement Learning with Verifiable Rewards (RLVR) or On-Policy Self-Distillation (OPSD).

By Xingjian Wu, Junlin Liu, Xingchen Liu, Xuhang Zhu, Jianing Wang, Linsen Guo, Xiaoyu Li, Xuezhi Cao, Xunliang Cai

llms agents reinforcement-learning efficiency benchmarks safety

arXiv Machine Learning

Jul 31

Enhancing Irregular Time Series Forecasting with Continuous-Time Modeling Framework

arXiv:2607. 28035v1 Announce Type: new Abstract: Irregular multivariate time series are widely encountered in applications such as healthcare monitoring, human activity recognition, and environmental sensing.

By Tianen Shen, Zhengyu Li, Yutong Li, Xiangfei Qiu, Xingjian Wu, Bin Yang, Jilin Hu

llms diffusion nlp benchmarks

arXiv Machine Learning

Jul 31

ClawTrack: Towards Trace-Level Evaluation and Improvement of Real-World Autonomous Agents

arXiv:2607. 28037v1 Announce Type: new Abstract: As LLM-based agents are deployed in complex, multi-step workflows, a critical evaluation gap has emerged: most existing benchmarks judge only final outcomes, unable to distinguish reliable reasoning from lucky success or attribute failures to specific process deficiencies, hindering attribution in long-horizon tasks.

By Xingjian Wu, Xuhang Zhu, Xingchen Liu, Junlin Liu, Jianing Wang, Linsen Guo, Xiaoyu Li, Xuezhi Cao, Xunliang Cai

llms agents benchmarks safety

arXiv Machine Learning

Jul 31

Chem World: A Large-Scale Benchmark and Physics-Informed Framework for Trustworthy Chemical Property Prediction

arXiv:2607. 28079v1 Announce Type: new Abstract: Chemical property prediction plays a critical role in accelerating scientific discovery in chemistry, materials science, and drug development.

By Tianyou Bai, Huan Wang, Mingchen Gao, Fangyue Lin, Pinze Ren, Zhenlin Zhao, Siming Dong

benchmarks

arXiv Machine Learning

Jul 31

LM-GRASP: Instance-Specific Language Models for Combinatorial Construction via Online Imitation Learning

arXiv:2607. 28135v1 Announce Type: new Abstract: Machine learning for combinatorial optimization typically relies on neural constructors trained via reinforcement learning on large offline datasets for a fixed problem class-incurring high pretraining costs and generalizing poorly outside the training distribution.

By Mohand Mezmaz, Gr\'egoire Danoy

llms reinforcement-learning benchmarks

arXiv Machine Learning

Jul 31

HARGO: Heterogeneity-Aware Reward-Guided Optimization for RL Post-Training of LLMs on HPC Tasks

arXiv:2607. 28301v1 Announce Type: new Abstract: Supervised fine-tuning (SFT) can equip large language models (LLMs) with domain knowledge for high-performance computing (HPC) tasks such as data race detection and benchmark question answering.

By Tiangang Li, Xiangbo Tian

llms nlp reinforcement-learning fine-tuning benchmarks safety

arXiv Machine Learning

Jul 31

Why Are GUI Agents Correct but Late? Decode on the Decision-Time Critical Path, Tested with Pre-Compiled Policy Trees

arXiv:2607. 28399v1 Announce Type: new Abstract: Computer-use agents often fail on transient GUI events because they produce the correct action only after the relevant window has already closed.

By Zihan Dong, Rui Qian, Qishi Zhan, Dongshen Peng, Kaixin Li, Yu Li

agents multimodal benchmarks

arXiv Machine Learning

Jul 31

$\beta$-OPSD: Deriving with Policy Optimization, Training with Self-Distillation

arXiv:2607. 28582v1 Announce Type: new Abstract: On-policy self-distillation (OPSD) is a promising approach to improve reasoning language models, but it remains brittle in practice: making it work reliably often requires substantial engineering effort.

By Jiawei Xu, Minghui Liu, Juzheng Zhang, Tom Goldstein, Furong Huang

llms reinforcement-learning efficiency benchmarks

Get the daily briefing