Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

21,071 stories · RSS feed

arXiv AI

Jun 24

CineCap: Structured Reasoning with Spatio-Temporal Anchors for Cinematographic Video Captioning

arXiv:2606. 24636v1 Announce Type: new Abstract: Cinematographic captioning aims to describe how a video is filmed using professional film-language concepts such as camera movement, shot size, depth of field, composition, and shooting angle.

By Xinyu Mao, Yuhui Zeng, Xiaokun Liu, Wenyu Qin, Meng Wang, Xin Tao, Pengfei Wan, Xiaohan Xing, Max Meng

llms diffusion reinforcement-learning fine-tuning multimodal benchmarks

arXiv Machine Learning

Jun 24

KLip-PPO: A per-sample KL perspective on PPO-Clip

arXiv:2606. 23932v1 Announce Type: new Abstract: Proximal Policy Optimization (PPO) is the standard policy-gradient algorithm for on-policy reinforcement learning.

By Riccardo Colletti, Robin Holzinger

reinforcement-learning benchmarks

arXiv Machine Learning

Jun 24

Flow-Corrected Thompson Sampling for Non-Stationary Contextual Bandits

arXiv:2606. 23933v1 Announce Type: cross Abstract: We study non-stationary linear contextual bandits where the reward model drifts over time, rendering classical contextual bandit algorithms brittle because historical data becomes systematically biased.

By AmirHossein Naghdi, Ali Baheri

reinforcement-learning benchmarks

arXiv Machine Learning

Jun 24

Real vs. Complex Spectral Bases for Neural Operators: The Role of Green's Function Alignment

arXiv:2606. 24851v1 Announce Type: new Abstract: Fourier Neural Operators (FNO) learn solution operators of partial differential equations by parameterizing global convolutions in the complex Fourier domain.

By Jason Sulskis, Sathya Ravi

benchmarks safety

arXiv AI

Jun 24

Legal Reasoning Is Not Lawyering: Rethinking Legal Benchmarks for Pro Se Access to Justice

arXiv:2606. 23716v1 Announce Type: cross Abstract: Legal AI benchmark research frequently invokes the assumption that large language models can improve access to justice, including for people who cannot access lawyers in order to understand and exercise their legal rights.

By Andrew Lou, David Shin

llms benchmarks safety

arXiv Machine Learning

Jun 24

BehaviorBench: Benchmarking Foundation Models for Behavioral Science Tasks

arXiv:2606. 24162v1 Announce Type: cross Abstract: Foundation models have been increasingly applied to behavioral science domains such as psychology, sociology, and economics.

By Jin Huang, Yutong Xie, Wanli Song, Xingjian Zhang, Walter Yuan, Matthew O. Jackson, Qiaozhu Mei

fine-tuning benchmarks safety

arXiv AI

Jun 24

Selective Capability Unlearning in End-to-End Spoken Language Understanding

arXiv:2606. 24063v1 Announce Type: cross Abstract: Modern spoken language understanding (SLU) systems are increasingly deployed in real-world settings, where specific functionalities may need to be removed due to policy or safety constraints.

By Akanksha Singh, Vinod Kumar Kurmi

benchmarks safety

arXiv AI

Jun 24

Assessing Distribution Shift in Human Activity Recognition for Domain Generalization

arXiv:2606. 24781v1 Announce Type: new Abstract: While the field of Human Activity Recognition (HAR) continues to draw interest from researchers and advance in important ways, some key challenges remain.

By Rebecca Adaimi, Edison Thomaz

benchmarks

arXiv AI

Jun 24

On the Stability of Prompt Ranking in Large Language Model Evaluation

arXiv:2606. 24381v1 Announce Type: cross Abstract: Prompt-based interaction has become a dominant paradigm for using large language models (LLMs), where multiple candidate prompts are evaluated and the top-ranked one is selected for downstream use.

By Shaoshuai Du, Penghao Liang, Yixian Shen, Chuanqi Shi, Hang Zhang, Lun Wang

llms benchmarks

arXiv Machine Learning

Jun 24

3D Masked Autoencoders are Robust Learners of Volumetric and Multimodal Cellular Representations for Microscopy

arXiv:2606. 23964v1 Announce Type: new Abstract: Self-supervised learning in fluorescence microscopy often relies on 2D projections, despite the inherently three-dimensional nature of cells.

By Amirhossein Kardoost, Lion Gleiter, Tingying Peng, Carsten Marr

llms multimodal benchmarks safety

arXiv Machine Learning

Jun 24

Target-Aware Linear Regression Under Distribution Shift

arXiv:2606. 22775v2 Announce Type: replace-cross Abstract: Distribution shift between training and deployment is a pervasive challenge for modern AI systems.

By Zhewen Hou, Tian Zheng

benchmarks safety

arXiv AI

Jun 24

Accuracy and Satisfaction in Multi-Turn LLM Dialogues for NFR Assessment

arXiv:2606. 24834v1 Announce Type: new Abstract: LLM-based dialogue assistants have become mainstream tools for software developers, yet current evaluation benchmarks focus exclusively on functional correctness.

By Ali Pourghasemi Fatideh, Wilder Baldwin, Maria Dhakal, Collin McMillan, Sepideh Ghanavati

llms agents benchmarks

arXiv AI

Jun 24

SAFARI: Scaling Long Horizon Agentic Fault Attribution via Active Investigation

arXiv:2606. 24626v1 Announce Type: new Abstract: As autonomous agents tackle increasingly complex multi-step, multi-agent tasks, their execution trajectories have scaled beyond the constraints of even the largest context windows.

By Chenyang Zhu, Jiayu Yao, Kushal Chawla, Youbing Yin, Nathan Wolfe, Pengshan Cai, Jingyu Wu, Spencer Hong, Sangwoo Cho, Shi-Xiong Zhang, Daben Liu, Sambit Sahu, Erin Babinsky

llms agents benchmarks

arXiv Machine Learning

Jun 24

Machine Learning Modeling for Real-Time Melt Pool Monitoring in Laser Powder Bed Fusion Additive Manufacturing: A Hybrid Approach

arXiv:2606. 23851v1 Announce Type: new Abstract: This work investigates the implementation of artificial intelligence and machine learning (AI/ML) for real-time monitoring in laser powder bed fusion (LPBF) additive manufacturing.

By Inioluwa Emmanuel, Zhuo Yang, Ho Yeung, Xinyao Zhang

rag computer-vision fine-tuning efficiency benchmarks

arXiv AI

Jun 24

Structural Kolmogorov-Arnold Convolutions: Learnable Function on the Values or the Filter Shape as Parameter-Efficient Alternative to Per-Edge Convolutional KANs

arXiv:2606. 24371v1 Announce Type: cross Abstract: Convolutional Kolmogorov--Arnold Networks (KANs) replace the fixed weights of a convolutional kernel with learnable univariate functions.

By Stefano Mereu, Oleksandr Kuznetsov, Gabriele Marchello, Alessandro Galdelli, Emanuele Frontoni, Adriano Mancini, Ferdinando Cannella

benchmarks

arXiv AI

Jun 24

The African Language Tax: Quantifying the Cost, Latency, and Context Penalty of Tokenizing African Languages in Frontier LLMs

arXiv:2606. 24460v1 Announce Type: cross Abstract: Commercial large language models bill, scale latency, and budget context per token.

By Olaoye Anthony Somide

llms nlp benchmarks

arXiv AI

Jun 24

AI-Driven Predictive Maintenance with Environmental Context Integration for Connected Vehicles: Simulation, Benchmarking, and Field Validation

arXiv:2603. 13343v3 Announce Type: replace-cross Abstract: Predictive maintenance for connected vehicles offers the potential to reduce unexpected breakdowns and improve fleet reliability, but most existing systems rely exclusively on internal diagnostic signals and are validated on simulated or industrial benchmark data.

By Kushal Khemani (Independent Researcher, India), Anjum Nazir Qureshi (Rajiv Gandhi College of Engineering Research,Technology)

fine-tuning benchmarks

arXiv AI

Jun 24

When Language Overwrites Vision: Over-Alignment and Geometric Debiasing in Vision-Language Models

arXiv:2605. 08245v4 Announce Type: replace-cross Abstract: Vision-Language Models (VLMs) increasingly power high-stakes applications, from medical imaging to autonomous systems, yet they routinely hallucinate, confidently describing content not present in the input.

By Harshvardhan Saini, Samyak Jha, Yiming Tang, Dianbo Liu

llms rag agents fine-tuning multimodal benchmarks safety

arXiv AI

Jun 24

Accelerating Disaggregated RL for Visual Generative LLMs with Diffusion-Based Parallelism and Trainer-Assisted Generation

arXiv:2606. 24369v1 Announce Type: new Abstract: Reinforcement learning (RL) has become a dominant post-training paradigm, driving the emergence of high-performance RL systems such as veRL for autoregressive large language models (LLMs).

By Sijie Wang, Zhengyu Qing, Zhiqiang Tan, Yiming Yin, Yeqing Zhang, Yaoyuan Wang, Qiang Wang, Xiaowen Chu, Shaohuai Shi

llms diffusion reinforcement-learning efficiency benchmarks

arXiv AI

Jun 24

Ensemble Distributionally Robust Bayesian Optimisation with Continuous Context

arXiv:2605. 07565v2 Announce Type: replace-cross Abstract: We study Bayesian Optimisation (BO) in settings where the objective function is influenced by uncontrollable environmental contexts governed by an unknown probability distribution.

By Tigran Ramazyan, Denis Derkach

benchmarks

Get the daily briefing