Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

22,941 stories · RSS feed

arXiv Machine Learning
Jun 9

Towards Personalized Bangla Book Recommendation: A Large-Scale Heterogeneous Book Graph Dataset

arXiv:2602. 12129v2 Announce Type: replace-cross Abstract: Personalized book recommendation in Bangla literature has been constrained by the lack of structured, large-scale, and publicly available datasets.

By Rahin Arefin Ahmed, Md. Anik Chowdhury, Sakil Ahmed Sheikh Reza, Devnil Bhattacharjee, Muhammad Abdullah Adnan, Julian McAuley, Nafis Sadeq
arXiv AI
Jun 9

Bayesian Selective Latent Inference for Wastewater-First Influenza Monitoring

arXiv:2606. 09433v1 Announce Type: new Abstract: Wastewater influenza surveillance can reveal community circulation before clinical reporting, but wastewater alone is not a fully identifiable proxy for human burden.

By Yixuan Zhang (Section of Health Data Science and AI, Department of Public Health, University of Copenhagen, Copenhagen, Denmark), Yang Song (Section of Health Data Science and AI, Department of Public Health, University of Copenhagen, Copenhagen, Denmark), Hao Wang (Rutgers University, New Brunswick, NJ, USA), Samir Bhatt (Section of Health Data Science and AI, Department of Public Health, University of Copenhagen, Copenhagen, Denmark, MRC Centre for Global Infectious Disease Analysis, Department of Infectious Disease Epidemiology, School of Public Health, Faculty of Medicine, Imperial College London, London, United Kingdom), Hengguan Huang (Section of Health Data Science and AI, Department of Public Health, University of Copenhagen, Copenhagen, Denmark, MRC Centre for Global Infectious Disease Analysis, Department of Infectious Disease Epidemiology, School of Public Health, Faculty of Medicine, Imperial College London, London, United Kingdom)
arXiv AI
Jun 9

REFLECT: Intervention-Supported Error Attribution for Silent Failures in LLM Agent Traces

arXiv:2606. 09071v1 Announce Type: new Abstract: Large language model (LLM) agents now solve complex tasks through long plan-and-execution traces, yet the ability to locate errors in a completed traces still lags far behind, especially in the \emph{silent failure} regime.

By Xiaofeng Lin, Yingxu Wang, Tung Sum Thomas Kwok, Daniel Guo, Sahil Arun Nale, Charles Fleming, Guang Cheng
arXiv AI
Jun 9

VFEM: Visual Feature Empowered Multivariate Time Series Forecasting with Cross-Modal Fusion

arXiv:2510. 03244v2 Announce Type: replace-cross Abstract: Large time series foundation models often adopt channel-independent architectures to handle varying data dimensions, but this design ignores crucial cross-channel dependencies.

By Yanlong Wang, Hang Yu, Jian Xu, Fei Ma, Hongkang Zhang, Tongtong Feng, Zijian Zhang, Shao-Lun Huang, Danny Dongning Sun, Xiao-Ping Zhang
arXiv Machine Learning
Jun 9

Hardware-aware Low-latency Quantum Compilation with Data-driven Lightweight Error Detection for Early Fault-Tolerant Systems

arXiv:2606. 07666v1 Announce Type: cross Abstract: Noisy intermediate-scale quantum (NISQ) processors are entering an early fault-tolerance regime where full quantum error correction carries prohibitive resource costs, yet lightweight error detection can meaningfully improve algorithmic success rates.

By Sumit Chongder (Indian Institute of Technology Jodhpur)
arXiv AI
Jun 9

A Dataset for Dynamic Human Preferences for Vision Language Models

arXiv:2606. 07653v1 Announce Type: cross Abstract: Given the increased adoption of Vision Language Models (VLMs) in human-interactive settings, it is important that we evaluate how well these models can adapt to real-time preferences for different users.

By Hannah Gao (Massachusetts Institute of Technology), Dylan Hadfield-Menell (Massachusetts Institute of Technology), Rachel Ma (Massachusetts Institute of Technology)
arXiv AI
Jun 9

VideoGPA: Distilling Geometry Priors for 3D-Consistent Video Generation

arXiv:2601. 23286v4 Announce Type: replace-cross Abstract: While recent video diffusion models (VDMs) produce visually impressive results, they fundamentally struggle to maintain 3D structural consistency, often resulting in object deformation or spatial drift.

By Hongyang Du, Junjie Ye, Xiaoyan Cong, Runhao Li, Jingcheng Ni, Aman Agarwal, Zeqi Zhou, Zekun Li, Randall Balestriero, Yue Wang
arXiv AI
Jun 9

Condition-Gated Reasoning for Context-Dependent Biomedical Question Answering

arXiv:2602. 17911v3 Announce Type: replace-cross Abstract: Current biomedical question answering (QA) systems often assume that medical knowledge applies uniformly, yet real-world clinical reasoning is inherently conditional: nearly every decision depends on patient-specific factors such as comorbidities and contraindications.

By Jash Rajesh Parekh, Wonbin Kweon, Joey Chan, Rezarta Islamaj, Robert Leaman, Pengcheng Jiang, Chih-Hsuan Wei, Zhizheng Wang, Zhiyong Lu, Jiawei Han
arXiv AI
Jun 9

Benchmarking Vision-Language-Action Models on SO-101: Failure and Recovery Analysis

arXiv:2606. 08881v1 Announce Type: cross Abstract: Vision-Language-Action (VLA) models have demonstrated strong generalization in robotic manipulation, yet existing evaluations are primarily conducted in simulation or on expensive robotic platforms, leaving their robustness on affordable real-world robots largely unexplored.

By Yi Yu, Xinchuan Qiu
arXiv AI
Jun 9

Integrating Deep Learning Demand Forecasting with Multi-Objective Optimization for Circular Coffee Supply Chains: A Data-Driven Framework for Cost, Emissions, and Freshness Management

arXiv:2606. 08314v1 Announce Type: new Abstract: The coffee supply chain is one of the most complex agri-food networks, marked by geographically dispersed production, multi-tier coordination, and high sensitivity to quality and freshness.

By Ger\c{c}ek Budak (Department of Industrial Engineering, Ankara Y{\i}ld{\i}r{\i}m Beyaz{\i}t University, Ke\c{c}i\"oren, Ankara 06010, T\"urkiye), Faraz Gholamzadeh Gharehgheshlaghi (Department of Industrial Engineering, Ankara Y{\i}ld{\i}r{\i}m Beyaz{\i}t University, Ke\c{c}i\"oren, Ankara 06010, T\"urkiye), Melika Barjesteh Vaezi (Department of Kinesiology and Sport Management, Texas Tech University, Lubbock, TX, United States), Ahmad Gholizadeh Lonbar (Department of Civil, Construction, and Environmental Engineering, University of Alabama, Tuscaloosa, AL, USA)