Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

24,477 stories · RSS feed

arXiv Machine Learning

Jun 5

Agents' Last Exam

arXiv:2606. 05405v1 Announce Type: cross Abstract: Recent AI systems have achieved strong results on a wide range of benchmarks, yet these gains have not translated into economically meaningful deployment across many professional domains.

By Yiyou Sun, Xinyang Han, Weichen Zhang, Yuanbo Pang, Tianyu Wang, Yuhan Cao, Yixiao Huang, Chris Duroiu, Haoyun Zhang, Jeffrey Lin, Weishu Zhang, Tyler Zeng, Ying Yan, Bo Liu, Hanson Wen, Mingyang Xu, Xiaoyuan Liu, Zimeng Chen, Weiyan Shi, Amanda Dsouza, Vincent Sunn Chen, Patrick Bryant, Carl Boettiger, Yamini Rangan, Bradley Rothenberg, Kyle Steinfeld, Arvind Rao, Tapio Schneider, Georgios Yannakakis, Laure Zanna, Kaan Ozbay, Ida Sim, Tarek Zohdi, George Em Karniadakis, Jack Gallant, Teresa Head-gordon, Yushan Li, Wenxi Deng, Tao Sun, Huiqi Wang, Zhun Wang, Justin Xu, Chris Yuhao Liu, Yafei Cheng, Rongwang Hu, Aras Bacho, Shengcao Cao, Zengyi Qin, Yixiong Chen, Hengduan Fan, Hao Liu, Lin Zeng, Shashank Muralidhar Bharadwaj, Litian Gong, Yingxuan Yang, Maojia Song, Ruheng Wang, Zongzheng Zhang, Honglin Bao, Shuo Lu, Jianhong Tu, Zhonghua Wang, Zheng Zhang, Zijiao Chen, yanqiong Jiang, Zhendong Li, Bohan Lyu, Chang Ma, Peiran Xu, Benran Zhang, Shangding Gu, Haoyue Hua, Haoyang Li, Wanzhe Liao, Chengzhi Liu, Junbo Peng, Haoran Sun, Zechen Xu, Bo Chen, Jiayi Cheng, Yi Jiang, Keying Kuang, Yuan Li, Youbang Pan, Ziyan Rao, Alexander Schubert, Yifan Shen, Vincent Siu, Xiatao Sun, Kangqi Zhang, Xiaopan Zhang, Yuchen Zhu, Ishaan Singh Chandok, Lei Ding, Jingxuan Fan, Andrew Glover, Jiaming Hu, Yiran Hu, Wenbo Huang, Zixin Jiang, Haoran Jin, Lukas Kim, Ming Liu, Yang Liu, Alireza Rafiei, Xuhuan Shen, Kunyang Sun, Sophia Sun, Ting Sun, Eric Wang, Yixin Wang, Hanwen Xing, Sihan Xu, Yuzheng Xu, Zhongxing Xu, Zhiling Yan, Boqin Yuan, Ruiqi Zhang, Yifan Zhang, Zibo Zhao, Liana, Santanu Bosu Antu, Haoyue Bai, Carlo Bosio, Joseph Cavanagh, Patricia Cavazos-Rehg, Tianxing Chen, Xuewen Chen, Yipu Chen, Zhu Chenyu, Chen Dai, Stefano De Castro, Yunfu Deng, Kaustubh Dhole, Jiayuan Ding, Chenchen Du, Zhehang Du, Hao Fan, Run-ze Fan, Hengyu Fu, Shi Gu, Yifan Gu, Charlie Guo, Baihe Huang, Baixiang Huang, Rimika Jaiswal, Zhihan Jiang, Ran Jin, Erin Kasson, Xin Lan, Joseph Lee, Deren Lei, Chenyu Li, Daofeng Li, Haitao Li, Hongwei Li, Jingyan Li, Xiao Li, Yi Li, Yinsheng Li, Yuangang Li, Zhixu Li, Wenyu Liang, Longtai Liao, Kevin Qinghong Lin, AndyZeyi Liu, Che Liu, Jiaming Liu, Kaiyuan Liu, Xuan Liu, Pan Lu, Wenbo Lv, Yicheng Lv, Qiuyang Mang, Kyle Montgomery, Yuzhou Nie, Ruoxi Ning, Jorin Overwiening, Xu Pan, Layna Paraboschi, Core Francisco Park, Justin Purnomo, Swati Rajwal, Scott Rankin, Bixuan Ren, Yiren Rong, HaoYang Shang, Ventus Shaw, Fiona Shen, Jiawei Shen, Minqi Shi, Qiu Shi, Huaxiu Yao, Tianneng Shi, Jonah So, Vladislav Susoy, Hannah Szlyk, Haocheng Wang, Jialu Wang, Wei Wang, Xinyu Wang, Zehao Wang, Dowling Wong, Angela Wu, Dehao Wu, Fangyu Wu, Mengyuan "Millie" Wu, Yu Wu, Yuchen Wu, Yuhao Wu, Qingpo Wuwu, Weihang Xiao, Yongyi Xiong, Fan Xu, Ruiling Xu, Mingxuan Yan, Benjamin Yang, Jirong Yang, Sen Yang, Xiaoli Yang, Yushi Yang, Haoran Ye, Xiaohu Yu, Zhengming Yu, Chenlong Zhang, Chi Zhang, Hanning Zhang, Hanwen Zhang, Junge Zhang, Kunpeng Zhang, Song Zhang, Wenjin Zhang, Wenshuo Zhang, Ying Zhang, Yizhi Zhang, Brian Zhao, Qijian Zhao, Yimin Zhao, Yuhaohua Zheng, Liwei Zhou, Tianyue Zhou, Sichen Zhu, Siqi Zhu, Yan Zhu, Yishu Zhu, Jierui Zuo, Chonghao Cai, Helena Casademunt, Wenjia Chen, Benjamin Cheng, Nawen Deng, Rao Fu, Tianfu Fu, Yifan Han, Ren He, Zhenyu He, Qiao Jin, Lang Lang, Yuetai Li, Sylvia Liu, Lu Lu, Qing Lu, Subhabrata Mukherjee, Yunqi Ouyang, Yin Ren, Dawei Shi, Haoran Wu, Zhiyue Wu, Hannah Yao, Zhuoran Yi, Jenny Yu, Rhea Zhan, Hang Zhou, Blake Zhu, Junfan Zhu, Alan Yuille, Yang Liu, Russell Alan Poldrack, Jiachen Li, Zhenglu Li, Molei Tao, Jing Huang, Wenqi Shi, Costas Spanos, Lichao Sun, Chenguang Wang, Orson Xu, Zhen Dong, Hector Gomez, Aylin Caliskan, Ali Emami, Haimin Hu, Zhi Li, Lihui Liu, Murphy Niu, Yi Shao, Jianxin Sun, Mikko Tolonen, Ting Wang, Sanjiv Das, Yanjun Gao, Wenbo Guo, Erika J Schneider, Zhiyong Lu, Mark Mueller, Radha Poovendran, Somayeh Sojoudi, Dawn Song

agents benchmarks

arXiv Machine Learning

Jun 5

Latent Reasoning with Normalizing Flows

arXiv:2606. 06447v1 Announce Type: cross Abstract: Large language models often improve reasoning by generating explicit chain-of-thought (CoT), demonstrating the importance of intermediate computation.

By Guancheng Tu, Xiangjun Fu, Suhao Yu, Yao Tang, Haoqiang Kang, Lianhui Qin, Yizhe Zhang, Jiatao Gu

llms efficiency benchmarks

arXiv Machine Learning

Jun 5

A New Quaternion-Joint Cable-Driven Redundant Manipulator Configuration and its Control Through FABRIK and Residual Reinforcement Learning

arXiv:2606. 05236v1 Announce Type: cross Abstract: Robotic arms capable of traversing arbitrary spatial paths, especially in highly obstructed workspaces, are highly desired across several industries.

By Tanapath Pornthisan, Thanapat Kemthong, Thanyapisit Kangsathien, Pasut Aranchaiya, Paulo Garcia, Viboon Sangveraphunsiri

reinforcement-learning robotics benchmarks

arXiv Machine Learning

Jun 5

Multimarginal flow matching with optimal transport potentials

arXiv:2606. 05327v1 Announce Type: new Abstract: Flow matching (FM) has emerged as a powerful framework for learning dynamic transport maps between two empirical distributions.

By Raghav Kansal, David Crair, Nghia Nguyen, Scott Pope, Bradley Parry

diffusion benchmarks

arXiv Machine Learning

Jun 5

SHRED: Retain-Set-Free Unlearning via Self-Distillation with Logit Demotion

arXiv:2605. 07482v2 Announce Type: replace Abstract: Machine unlearning for large language models (LLMs) aims to selectively remove memorized content such as private data, copyrighted text, or hazardous knowledge, without costly full retraining.

By Zizhao Hu, Ameya Godbole, Johnny Tian-Zheng Wei, Mohammad Rostami, Jesse Thomason, Robin Jia

llms efficiency benchmarks

arXiv Machine Learning

Jun 5

Query-efficient model evaluation using cached responses

arXiv:2605. 07096v2 Announce Type: replace Abstract: Evaluating a new model on an existing benchmark is often necessary to understand its behavior before deployment.

By Hayden Helm, Ben Johnson, Carey Priebe

benchmarks

arXiv Machine Learning

Jun 5

FATE: Focal-modulated Attention Encoder for Multivariate Time-series Forecasting

arXiv:2408. 11336v3 Announce Type: replace Abstract: Climate change stands as one of the most pressing global challenges of the twenty-first century, with far-reaching consequences such as rising sea levels, melting glaciers, and increasingly extreme weather patterns.

By Tajamul Ashraf, Janibul Bashir

llms benchmarks safety

arXiv Machine Learning

Jun 5

Stochastic-Dimension Frozen Sampled Neural Network for High-Dimensional Gross-Pitaevskii Equations on Unbounded Domains

arXiv:2604. 09361v3 Announce Type: replace Abstract: This paper introduces the Stochastic-Dimension Frozen Sampled Neural Network (SD-FSNN), a novel computational framework for solving high-dimensional Gross-Pitaevskii equation (GPE) on unbounded domain.

By Zhangyong Liang, Tingfeng Wang, Xiaofei Zhao

benchmarks

arXiv Machine Learning

Jun 5

Reframing preprocessing selection as model-internal calibration in near-infrared spectroscopy: A large-scale benchmark of operator-adaptive PLS and Ridge models

arXiv:2605. 13587v3 Announce Type: replace-cross Abstract: Preprocessing screening is often the most expensive part of a near-infrared spectroscopy calibration workflow.

By Gregory Beurier (CIRAD, UMR AGAP Institut, Montpellier, France, UMR AGAP Institut, Univ Montpellier, CIRAD, INRAE, Institut Agro, Montpellier, France), Robin Reiter (CIRAD, UMR AGAP Institut, Montpellier, France, UMR AGAP Institut, Univ Montpellier, CIRAD, INRAE, Institut Agro, Montpellier, France), Camille No\^us (Laboratoire Cogitamus), Lauriane Rouan (CIRAD, UMR AGAP Institut, Montpellier, France, UMR AGAP Institut, Univ Montpellier, CIRAD, INRAE, Institut Agro, Montpellier, France), Denis Cornet (CIRAD, UMR AGAP Institut, Montpellier, France, UMR AGAP Institut, Univ Montpellier, CIRAD, INRAE, Institut Agro, Montpellier, France)

benchmarks

arXiv Machine Learning

Jun 5

Plug-and-Play Guidance for Discrete Diffusion Models via Gradient-Informed Logit Correction

arXiv:2606. 06303v1 Announce Type: new Abstract: Controllable generation with discrete diffusion models is often hindered by high computational overhead or the need for retraining.

By Hongkun Dou, Zike Chen, Fengji Li, Hongjue Li, Yue Deng

diffusion fine-tuning benchmarks

arXiv Machine Learning

Jun 5

Training One Model to Master Cross-Level Agentic Actions via Reinforcement Learning

arXiv:2512. 09706v2 Announce Type: replace Abstract: The paradigm of agentic AI is shifting from engineered complex workflows to post-training native models.

By Kaichen He, Zihao Wang, Muyao Li, Anji Liu, Yitao Liang

agents reinforcement-learning robotics fine-tuning benchmarks

arXiv Machine Learning

Jun 5

Spectral Probe-Circuits: A Three-Step Recipe for Identifying Attention-Head Circuits in Pretrained Transformers

arXiv:2605. 24059v2 Announce Type: replace Abstract: We present a three-step recipe for identifying attention-head circuits in pretrained transformers.

By Yongzhong Xu

llms benchmarks

arXiv Machine Learning

Jun 5

When Attention Beats Fourier: Multi-Scale Transformers for PDE Solving on Irregular Domains

arXiv:2605. 08318v2 Announce Type: replace Abstract: We study the problem of \emph{architecture selection} for deep learning models trained to solve partial differential equations (PDEs), asking when transformer-based architectures with learned attention outperform Fourier-domain neural operators.

By Brandon Yee, Pairie Koh, Jack Rodriguez, Mihir Tekal

llms diffusion benchmarks safety

arXiv Machine Learning

Jun 5

Uncovering Model Processing Strategies with Non-Negative Per-Example Fisher Factorization

arXiv:2310. 04649v3 Announce Type: replace Abstract: We introduce NPEFF (Non-Negative Per-Example Fisher Factorization), an interpretability method that aims to uncover strategies used by a model to generate its predictions.

By Michael Matena, Colin Raffel

llms benchmarks safety

arXiv Machine Learning

Jun 5

Agentic Monte Carlo: Simulating Reinforcement Learning for Black-Box Agents

arXiv:2606. 05296v1 Announce Type: new Abstract: LLM agents operate in two distinct regimes: open-weight agents amenable to reinforcement learning (RL) and black-box agents whose behaviour must be controlled purely at test time.

By Dae Yon Hwang, Raunaq Suri, Valentin Villecroze, Anthony L. Caterini, Jesse C. Cresswell, No\"el Vouitsis, Brendan Leigh Ross

llms agents reinforcement-learning benchmarks

arXiv Machine Learning

Jun 5

HOPSE: Scalable Higher-Order Positional and Structural Encoder for Combinatorial Representations

arXiv:2505. 15405v3 Announce Type: replace Abstract: While Graph Neural Networks (GNNs) have proven highly effective at modeling relational data, pairwise connections cannot fully capture multi-way relationships naturally present in complex real-world systems.

By Guillermo Bern\'ardez, Marco Montagna, Louis Van Langendonck, Martin Carrasco, Amirreza Akbari, Louisa Cornelis, Mathilde Papillon, Pere Barlet-Ros, Nina Miolane, Lev Telyatnikov

benchmarks

arXiv Machine Learning

Jun 5

MSTN: A Lightweight and Fast Model for General TimeSeries Analysis

arXiv:2511. 20577v5 Announce Type: replace Abstract: Real-world time series often exhibit strong non-stationarity, complex nonlinear dynamics, and behavior expressed across multiple temporal scales, from rapid local fluctuations to slow-evolving long-range trends.

By Sumit S Shevtekar, Chandresh K Maurya

llms nlp benchmarks

arXiv Machine Learning

Jun 5

The Post-GCN Decade Revisited: Curvature-Stratified Evaluation of Relational Learning

arXiv:2606. 06397v1 Announce Type: new Abstract: Current evaluation practices in relational learning rely heavily on flat leaderboards that average performance across heterogeneous datasets, implicitly assuming a uniform underlying structure.

By Shuo Wang, Xiangyu Wang, Quanxin Wang, Bailin Wu, Bokui Wang, Shunyang Huang, Boyan Deng, Haonan Liu, Ruiyi Fang, Zhenxiang Xu, Boyu Wang, Zhao Kang

benchmarks safety

arXiv Machine Learning

Jun 5

Step-adaptive multimodal fusion network with multi-scale cloud feature learning for ultra-short-term solar irradiance forecasting

arXiv:2606. 06102v1 Announce Type: cross Abstract: Ultra-short-term solar irradiance prediction is critical for photovoltaic system dispatch and power grid stability.

By Jingxin Zhang Xiaoqin Wang

multimodal benchmarks

arXiv Machine Learning

Jun 5

Is attention truly all we need? An empirical study of asset pricing in pretrained RNN sparse and global attention models

arXiv:2508. 19006v2 Announce Type: replace-cross Abstract: This study investigates the pre-trained RNN attention models with the mainstream attention mechanisms, such as additive attention, Luong's three attentions, global self-attention and sliding window sparse attention, for the empirical asset pricing research on the top 420 large-cap US stocks.

By Shanyan Lai

llms efficiency benchmarks

Get the daily briefing