Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

20,687 stories · RSS feed

arXiv AI

Jun 24

A Benchmark for Hallucination Detection in VLMs for Gastrointestinal Endoscopy

arXiv:2606. 24115v1 Announce Type: cross Abstract: Vision-language models (VLMs) are prone to hallucination, which remains a major barrier to their safe deployment in clinical practice.

By Aminu Lawal, Niyoj Oli, Sachin Acharya, Prashnna Gyawali, Maria Carmen Romano, Binod Bhattarai

llms nlp multimodal benchmarks safety

arXiv AI

Jun 24

Ten Digits on a Train: AI-Assisted Verification of Two Eigenvalue Problems

arXiv:2606. 23821v1 Announce Type: cross Abstract: Accurate numerical eigenvalues are often difficult to certify, especially in singular or non-normal settings.

By Matthew J. Colbrook

benchmarks

arXiv Machine Learning

Jun 24

Open-Vocabulary BEV Segmentation with 3D-Aware Geometric Constraints

arXiv:2606. 24353v1 Announce Type: cross Abstract: Bird's-eye view (BEV) perception fuses multi-camera images into a unified top-down representation for autonomous driving.

By Hojun Choi, Seulbin Hwang, Dae Jung Kim, Kisung Kim, Hyunjung Shim, Jinhan Lee

llms agents computer-vision robotics efficiency multimodal benchmarks

arXiv AI

Jun 24

GUI vs. CLI: Execution Bottlenecks in Screen-Only and Skill-Mediated Computer-Use Agents

arXiv:2606. 24551v1 Announce Type: new Abstract: Computer-use agents can execute software tasks through either graphical interfaces or programmatic command interfaces, but existing evaluations confound interaction modality with differences in tasks, initial states, verifiers, and permitted actions.

By Xiao Zhou, Siyue Zhang, Yilun Zhao, Jinbiao Wei, Tingyu Song, Arman Cohan, Chen Zhao

agents benchmarks

arXiv AI

Jun 24

Learning to Trigger: Reinforcement Learning at the Large Hadron Collider

arXiv:2606. 23993v1 Announce Type: cross Abstract: High-throughput scientific facilities such as the Large Hadron Collider depend on real-time event filtering (\textit{triggering}) under tight constraints on bandwidth, latency, and storage.

By Zixin Ding, Shaghayegh Emam, Giovanna Salvi, Cecilia Tosciri, Abhijith Gandrakota, Jennifer Ngadiuba, Nhan Tran, Christian Herwig, David W. Miller, Yuxin Chen

agents reinforcement-learning fine-tuning efficiency benchmarks

arXiv AI

Jun 24

RAVEN: A Regime-Aware Variable-context Expert Network for Financial Time Series Forecasting

arXiv:2606. 24062v1 Announce Type: cross Abstract: Financial time series forecasting presents structural challenges absent from standard benchmarks.

By Cheng He, Zhenyu Guan, Xijie Liang, Defu Lian, Jiajia Li, Enhong Chen, Patrick P. C. Lee, Geng Hu, Zehao Chen

llms benchmarks

arXiv AI

Jun 24

Alternate loss functions and regression models that achieve robustness to outliers by modulating the learning rate

arXiv:2606. 22068v2 Announce Type: replace-cross Abstract: Most real-world datasets used for training supervised learning models are contaminated with noisy data and outliers leading to large prediction errors.

By Mathew Mithra Noel, Arindam Banerjee, Yug D. Oswal, Geraldine Bessie Amali D, Venkataraman Muthiah-Nakarajan

benchmarks

arXiv AI

Jun 24

Offline Reinforcement Learning for Warehouse SLAM Throughput Control

arXiv:2606. 23978v1 Announce Type: cross Abstract: We present an offline reinforcement learning (RL) framework for optimizing SLAM throughput control in a warehouse fulfillment environment.

By Tina Dongxu Li, Mouhacine Benosman, Rajat Kumar, Kevin Tan, Ken Meszaros, Trevor Dardik

reinforcement-learning efficiency benchmarks

arXiv AI

Jun 24

IPO Finance Agent: Evaluation of LLM Financial Analysts beyond Finance Agent v2, with Automated Rubric Generation -- the Case of the SpaceX (SPCX) IPO

arXiv:2606. 23032v2 Announce Type: replace Abstract: Finance Agent v2 (by Vals AI) has emerged as the reference benchmark for evaluating both Anthropic Claude and OpenAI ChatGPT frontier language models on financial tasks.

By Mostapha Benhenda

llms agents benchmarks safety

arXiv AI

Jun 24

Attention in Motion: Secure Platooning via Transformer-based Misbehavior Detection

arXiv:2512. 15503v3 Announce Type: replace-cross Abstract: Vehicular platooning promises transformative improvements in transportation efficiency and safety through the coordination of multi-vehicle formations enabled by Vehicle-to-Everything (V2X) communication.

By Konstantinos Kalogiannis, Ahmed Mohamed Hussain, Hexu Li, Panos Papadimitratos

llms efficiency benchmarks safety

arXiv Machine Learning

Jun 24

A Physics-Informed Fourier-Wavelet Transformer for Multiscale Computational Fluid Dynamics Surrogate Modeling

arXiv:2606. 24696v1 Announce Type: cross Abstract: Physics-informed surrogate models can accelerate computational fluid dynamics simulations.

By Somyajit Chakraborty, Ming Pan, Xizhong Chen

llms benchmarks

arXiv AI

Jun 24

Benchmarking LLMs' Mathematical Reasoning with Unseen Random Variables Questions

arXiv:2501. 11790v5 Announce Type: replace-cross Abstract: Recent studies have raised significant concerns regarding the reliability of current mathematics benchmarks, highlighting issues such as simplistic design and potential data contamination.

By Zijin Hong, Hao Wu, Su Dong, Junnan Dong, Yilin Xiao, Yujing Zhang, Zhu Wang, Feiran Huang, Linyi Li, Hongxia Yang, Xiao Huang

llms benchmarks

arXiv AI

Jun 24

Event-Grounded Question Answering over Long Audio via Structured Retrieval

arXiv:2602. 14612v4 Announce Type: replace-cross Abstract: Answering natural-language questions over multi-hour audio requires both event recognition and temporal grounding.

By Kartik Hegde, Arvind Krishna Sridhar, Naveen Vakada, Yinyi Guo, Erik Visser

llms rag nlp benchmarks data-engineering

arXiv Machine Learning

Jun 24

Holistic Data Scheduler for LLM Pre-training via Multi-Objective Reinforcement Learning

arXiv:2606. 24133v1 Announce Type: new Abstract: The composition of training data, governed by the diversity of sources and their mixing strategy, is a cornerstone of Large Language Model (LLM) pre-training.

By Chenhao Dang, Jing Ma, Mingjie Liao

llms reinforcement-learning benchmarks

arXiv AI

Jun 24

Selective Capability Unlearning in End-to-End Spoken Language Understanding

arXiv:2606. 24063v1 Announce Type: cross Abstract: Modern spoken language understanding (SLU) systems are increasingly deployed in real-world settings, where specific functionalities may need to be removed due to policy or safety constraints.

By Akanksha Singh, Vinod Kumar Kurmi

benchmarks safety

arXiv AI

Jun 24

Accuracy and Satisfaction in Multi-Turn LLM Dialogues for NFR Assessment

arXiv:2606. 24834v1 Announce Type: new Abstract: LLM-based dialogue assistants have become mainstream tools for software developers, yet current evaluation benchmarks focus exclusively on functional correctness.

By Ali Pourghasemi Fatideh, Wilder Baldwin, Maria Dhakal, Collin McMillan, Sepideh Ghanavati

llms agents benchmarks

arXiv AI

Jun 24

Deciphering Fingerprints of 3D Molecular Surfaces for Accurate Epitope Prediction

arXiv:2606. 23830v1 Announce Type: cross Abstract: Molecular surfaces encode the geometric and physicochemical patterns that determine antibody-antigen recognition, central to epitope prediction.

By Fang Wu, Weihao Xuan, Jure Leskovec, Yejin Choi, Li Erran Li

llms benchmarks

arXiv AI

Jun 24

Mind the Heads: Topological Representation Alignment for Multimodal LLMs

arXiv:2606. 23885v1 Announce Type: cross Abstract: Representation alignment has emerged as an effective approach to improve Multimodal Large Language Models (MLLMs) by regularizing their internal representations toward those of an external vision encoder.

By Davide Caffagni, Alberto Compagnoni, Federico Melis, Sara Sarto, Pier Luigi Dovesi, Mark Granroth-Wilding, Marcella Cornia, Lorenzo Baraldi

llms multimodal benchmarks safety

arXiv AI

Jun 24

Blockwise Policy-Drift Gating for On-Policy Distillation

arXiv:2606. 24084v1 Announce Type: cross Abstract: On-policy distillation (OPD) trains a student policy using teacher signals computed on trajectories sampled by the student itself.

By Liwen Zheng, Haiyun Jiang

efficiency benchmarks

arXiv Machine Learning

Jun 24

Adversarial dynamical systems characterize when data-driven learning succeeds or fails

arXiv:2407. 06312v2 Announce Type: replace-cross Abstract: Many systems resist analytical modeling, making data-driven inference of dynamics important.

By Matthew J. Colbrook, Igor Mezi\'c, Alexei Stepanenko

benchmarks safety data-engineering

Get the daily briefing