Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

24,477 stories · RSS feed

arXiv Machine Learning

Jun 4

Hyper-ICL: Attention Calibration with Hyperbolic Anchor Distillation for Multimodal In-Context Learning

arXiv:2606. 04434v1 Announce Type: cross Abstract: Multimodal In-Context Learning (ICL) has emerged as a practical inference paradigm for Multimodal Large Language Models, where a small set of interleaved image-text In-Context Demonstrations (ICDs) conditions the model to solve new tasks.

By Niloufar Alipour Talemi, Hossein Kashiani, Fatemeh Afghah

llms efficiency multimodal benchmarks

arXiv AI

Jun 4

Proof-Carrying Agent Actions: Model-Agnostic Runtime Governance for Heterogeneous Agent Systems

arXiv:2606. 04104v1 Announce Type: cross Abstract: Agent systems execute through runtimes with very different control points: local coding tools, framework SDKs, managed agent platforms, API gateways, and observer-only integrations.

By Zexun Wang

agents benchmarks

arXiv Machine Learning

Jun 4

CRAFT: Cost-aware Refinement And Front-aware Tuning of Prompts

arXiv:2606. 04661v1 Announce Type: cross Abstract: Prompts tuned for accuracy often grow long, raising inference cost on every model call.

By Shanu Kumar, Shubhanshu Khandelwal, Akhila Yesantarao Venkata, Parag Agrawal, Yova Kementchedjhieva, Manish Gupta

llms benchmarks

arXiv Machine Learning

Jun 4

Bayesian learning for the stochastic shortest path problem

arXiv:2606. 04845v1 Announce Type: cross Abstract: Sequential decision-making problems are often modelled as a Markov decision process (MDP).

By Chon Wai Ho, Sumeetpal S. Singh, Jiaqi Guo

reinforcement-learning benchmarks

arXiv Machine Learning

Jun 4

An Open-Source Two-Stage Computer Vision Pipeline for Fine-Grained Vehicle Classification using Vision Transformers

arXiv:2606. 05149v1 Announce Type: cross Abstract: Vehicle body type is a significant determinant of cyclist injury severity in overtaking crashes, yet automated tools for classifying vehicles into injury-risk-relevant categories from naturalistic roadway video do not exist in the open literature.

By Gandhimathi Padmanaban, Fred Feng

llms computer-vision fine-tuning benchmarks safety

arXiv Machine Learning

Jun 4

Neural Variance-aware Dueling Bandits with Deep Representation and Shallow Exploration

arXiv:2506. 01250v3 Announce Type: replace Abstract: We introduce the first variance-aware algorithms for contextual dueling bandits that leverage shallow exploration strategies with neural networks for nonlinear utility approximation.

By Youngmin Oh, Jinje Park, Taejin Paik

benchmarks

arXiv Machine Learning

Jun 4

Identifying and Correcting Label Noise for Robust GNNs via Influence Contradiction

arXiv:2601. 17469v2 Announce Type: replace Abstract: Graph Neural Networks (GNNs) have shown remarkable capabilities in learning from graph-structured data with various applications such as social analysis and bioinformatics.

By Wei Ju, Wei Zhang, Siyu Yi, Zhengyang Mao, Yifan Wang, Jingyang Yuan, Zhiping Xiao, Ziyue Qiao, Ming Zhang

diffusion benchmarks

arXiv Machine Learning

Jun 4

On the Expressive Power of Permutation-Equivariant Weight-Space Networks

arXiv:2602. 01083v2 Announce Type: replace Abstract: Weight-space learning studies neural architectures that operate directly on the parameters of other neural networks.

By Adir Dayan, Yam Eitan, Haggai Maron

benchmarks

arXiv AI

Jun 4

Adaptive Patching Is Harder Than It Looks For Time-Series Forecasting

arXiv:2606. 04074v1 Announce Type: cross Abstract: Adaptive patching is a recent and compelling proposal for time-series Transformers: allocate finer patches where the sequence looks locally informative.

By Federico Zucchi, Yi Xie, Chao Zhang, Keyuan Luo, Thomas Lampert, Ziyue Li

llms benchmarks safety

arXiv AI

Jun 4

RUBAS: Rubric-Based Reinforcement Learning for Agent Safety

arXiv:2606. 04051v1 Announce Type: cross Abstract: The evolution of LLMs into tool-enabled agents creates a new class of safety challenges associated with real-world execution rather than simple text generation.

By Xian Qi Loye, Qinglin Su, Zhexin Zhang, Shiyao Cui, Qi Zhu, Fei Mi, Hongning Wang, Minlie Huang

llms agents reinforcement-learning benchmarks safety

arXiv Machine Learning

Jun 4

Breaking the Scale Barrier: One-Shot Knowledge Transfer via Frequency Transform

arXiv:2603. 07523v3 Announce Type: replace Abstract: Transferring knowledge by fine-tuning large-scale pre-trained networks has become a standard paradigm for downstream tasks, yet the knowledge of a pre-trained model is tightly coupled with monolithic architecture, which restricts flexible reuse across models of varying scales.

By Jianlu Shen, Fu Feng, Yucheng Xie, Jiaqi Lv, Xin Geng

fine-tuning benchmarks

arXiv Machine Learning

Jun 4

Towards interpretable AI with quantum annealing feature selection

arXiv:2604. 25649v2 Announce Type: replace Abstract: Deep learning models are used in critical applications, in which mistakes can have serious consequences.

By Francesco Aldo Venturelli, Emanuele Costa, Sikha O K, Bruno Juli\'a-D\'iaz, Miguel A. Gonz\'alez Ballester, Alba Cervera-Lierta

computer-vision benchmarks

arXiv Machine Learning

Jun 4

The Perception-Physics Paradox: Probing Scientific Alignment with TC-Bench

arXiv:2605. 24782v2 Announce Type: replace Abstract: While Vision Foundation Models (VFMs) excel at predictive tasks on satellite imagery, their performance can arise from visual correlations rather than underlying structural invariants, making even perception-based out-of-distribution accuracy a poor proxy for scientific utility.

By Dingling Yao, Andrea Polesello, Adeel Pervez, Caroline Muller, Francesco Locatello

benchmarks safety

arXiv Machine Learning

Jun 4

Flow Matching Calibration for Simulation-Based Inference under Model Misspecification

arXiv:2509. 23385v5 Announce Type: replace-cross Abstract: Simulation-based inference (SBI) is transforming experimental sciences by enabling parameter estimation in complex non-linear models from simulated data.

By Pierre-Louis Ruhlmann, Michael Arbel, Florence Forbes, Pedro L. C. Rodrigues

diffusion benchmarks

arXiv Machine Learning

Jun 4

Outcome-Grounded Advantage Reshaping for Fine-Grained Credit Assignment in Mathematical Reasoning

arXiv:2601. 07408v2 Announce Type: replace-cross Abstract: Group Relative Policy Optimization (GRPO) has emerged as a promising critic-free reinforcement learning paradigm for reasoning tasks.

By Ziheng Li, Liu Kang, Feng Xiao, Luxi Xing, Qingyi Si, Zhuoran Li, Weikang Gong, Deqing Yang, Yanghua Xiao, Hongcheng Guo

llms reinforcement-learning benchmarks

arXiv Machine Learning

Jun 4

Plug-and-Play Diffusion Meets ADMM: Dual-Variable Coupling for Robust Medical Image Reconstruction

arXiv:2602. 23214v2 Announce Type: replace-cross Abstract: Plug-and-Play diffusion prior (PnPDP) frameworks have emerged as a powerful paradigm for solving imaging inverse problems by treating pretrained generative models as modular priors.

By Chenhe Du, Xuanyu Tian, Qing Wu, Muyu Liu, Jingyi Yu, Hongjiang Wei, Yuyao Zhang

diffusion fine-tuning benchmarks safety

arXiv Machine Learning

Jun 4

Attention-Based Sampler for Diffusion Language Models

arXiv:2604. 08564v2 Announce Type: replace-cross Abstract: Auto-regressive models (ARMs) have established a dominant paradigm in language modeling.

By Yuyan Zhou, Kai Syun Hou, Weiyu Chen, James Kwok

llms diffusion benchmarks

arXiv Machine Learning

Jun 4

Neural Autoregressive Control Variates for the Quantum Monte Carlo Sign Problem

arXiv:2605. 26814v2 Announce Type: replace-cross Abstract: We train a pair of autoregressive models to construct zero-mean control variates to mitigate the sign problem in quantum Monte Carlo simulations.

By Bei Qiao, Lei Wang

llms benchmarks

arXiv AI

Jun 4

Inclusion-of-Thoughts: Mitigating Preference Instability via Purifying the Decision Space

arXiv:2604. 04944v2 Announce Type: replace-cross Abstract: Multiple-choice questions (MCQs) are widely used to evaluate large language models (LLMs).

By Mohammad Reza Ghasemi Madani, Soyeon Caren Han, Shuo Yang, Jey Han Lau

llms benchmarks safety

arXiv AI

Jun 4

PoliticsBench: Benchmarking Political Values in Large Language Models with Multi-Turn Roleplay

arXiv:2603. 23841v2 Announce Type: replace-cross Abstract: While Large Language Models (LLMs) are increasingly used as primary sources of information, their potential for political bias may impact their objectivity.

By Rohan Khetan, Ashna Khetan

llms benchmarks safety

Get the daily briefing