Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

22,941 stories · RSS feed

arXiv AI

Jun 9

SENTRY: Statistical Reliability Analysis of Vision Transformers Under Soft Errors

arXiv:2606. 07620v1 Announce Type: cross Abstract: With the growth of Vision Transformers in safety-critical domains like autonomous systems and medical imaging, ensuring their reliability against soft errors is paramount.

By Pramit Kumar Bhaduri, Mahdi Taheri, Samira Nazari, Maksim Jenihhin, Christian Herglotz, Michael Hubner

llms agents benchmarks safety

arXiv Machine Learning

Jun 9

Large-scale empirical tuning and comparison of default optimizers for variational inference

arXiv:2606. 07841v1 Announce Type: cross Abstract: Black-box variational inference (BBVI) is a methodology for posterior approximation that relies on stochastic optimization.

By Trevor Campbell, Jonathan H. Huggins, Kyurae Kim, Charles C. Margossian

benchmarks

arXiv AI

Jun 9

Zero-Shot Learning in Industrial Scenarios: New Large-Scale Benchmark, Challenges and Baseline

arXiv:2606. 07965v1 Announce Type: new Abstract: Large Visual Language Models (LVLMs) have achieved remarkable success in vision tasks.

By Zekai Zhang, Qinghui Chen, Maomao Xiong, Shijiao Ding, Zhanzhi Su, Xinjie Yao, Yiming Sun, Cong Bai, Jinglin Zhang

llms fine-tuning multimodal benchmarks

arXiv AI

Jun 9

Speech Enhancement Based on Drifting Models

arXiv:2604. 24199v4 Announce Type: replace-cross Abstract: We propose Speech Enhancement based on Drifting Models (DriftSE), a novel generative framework that formulates denoising as an equilibrium problem.

By Liang Xu, Diego Caviedes-Nozal, W. Bastiaan Kleijn, Longfei Felix Yan, Rasmus Kongsgaard Olsson

diffusion benchmarks

arXiv Machine Learning

Jun 9

Nonparametric LLM Evaluation from Preference Data

arXiv:2601. 21816v2 Announce Type: replace Abstract: Evaluating the performance of large language models (LLMs) from human preference data is crucial for obtaining LLM leaderboards.

By Dennis Frauen, Athiya Deviyani, Mihaela van der Schaar, Stefan Feuerriegel

llms benchmarks

arXiv AI

Jun 9

Voting Protocols as Coordination Mechanisms for Role-Constrained Multi-Agent Tutoring Systems

arXiv:2606. 08030v1 Announce Type: cross Abstract: Agentic tutoring systems introduce a coordination challenge: multiple agents may propose different but reasonable interventions, yet only one response can be delivered to the learner.

By Eric S. Qiu, Joyce Gill

agents benchmarks

arXiv Machine Learning

Jun 9

ForcingDAS: Unified and Robust Data Assimilation via Diffusion Forcing

arXiv:2605. 14285v2 Announce Type: replace-cross Abstract: Data assimilation (DA) estimates the state of an evolving dynamical system from noisy, partial observations, and is widely used in scientific simulation as well as weather and climate science.

By Yixuan Jia, Siyi Chen, Yida Pan, Xiao Li, Lianghe Shi, Chanyong Jung, Haijie Yuan, Ismail Alkhouri, Yue Cynthia Wu, Saiprasad Ravishankar, Jeffrey A Fessler, Qing Qu

diffusion benchmarks

arXiv AI

Jun 9

AHA-WAM:Asynchronous Horizon-Adaptive World-Action Modeling with Observation-Guided Context Routing

arXiv:2606. 09811v1 Announce Type: cross Abstract: World-action models have emerged as a promising paradigm for robot manipulation, jointly modeling visual scene dynamics and actions to inject physical priors into policy learning.

By Jisong Cai, Long Ling, Shiwei Chu, Zhongshan Liu, Jiayue Kang, Zhixuan Liang, Wenjie Xu, Yinan Mao, Weinan Zhang, Xiaokang Yang, Ru Ying, Ran Zheng, Yao Mu

llms diffusion robotics benchmarks

arXiv AI

Jun 9

Robust-U1: Can MLLMs Self-Recover Corrupted Visual Content for Robust Understanding?

arXiv:2606. 08063v1 Announce Type: cross Abstract: Multimodal Large Language Models (MLLMs) have demonstrated remarkable success in visual understanding, yet their performance degrades significantly under real-world visual corruptions.

By Jiaqi Tang, Jianmin Chen, Youyang Zhai, Wei Wei, Runtao Liu, Mengjie Zhao, Xiangyu Wu, Qingfa Xiao, Qifeng Chen

llms reinforcement-learning fine-tuning multimodal benchmarks safety

arXiv AI

Jun 9

A multi-agent system for spine MRI report generation from multi-sequence imaging

arXiv:2606. 08897v1 Announce Type: cross Abstract: Spinal pathology is a leading cause of pain and disability worldwide.

By Zhiping Xiao, Junwei Yang, Gongbo Sun, Han Zhang, Hanwen Xu, Yi Yao, Zachary D. Miller, William E. King III, Mohammed M. Kanani, Jalal B. Andre, Sammy Chu, Ming Zhang, Paul E. Kinahan, Nathan M. Cross, Sheng Wang

rag agents multimodal benchmarks

arXiv Machine Learning

Jun 9

Can LLMs extract scientific consensus? A case study in high-temperature superconductivity

arXiv:2606. 07570v1 Announce Type: cross Abstract: Scientific knowledge is increasingly dispersed across vast and heterogeneous scientific literature, where important claims are often implicit, evolving, and internally debated.

By Mouyang Cheng, Wenhao He, Zhuotao Jin, Bowen Yu, Ju Li, Boris Kozinsky, Yao Wang, Pavel Volkov, Liangzi Deng, Ching-Wu Chu, Xiao-Gang Wen, Mingda Li

llms nlp benchmarks

arXiv AI

Jun 9

Customer Churn Prediction on Structured Data Using FT-Transformer and Stacking Ensembles

arXiv:2606. 07582v1 Announce Type: cross Abstract: Customer churn prediction is essential across data-driven industries such as insurance, digital banking, eCommerce, and subscription platforms, where retaining existing customers is typically more cost-effective than acquiring new ones.

By Joyjit Roy, Samaresh Kumar Singh, Laxmi Shaw

llms benchmarks

arXiv AI

Jun 9

Sound and Complete Neurosymbolic Reasoning with LLM-Grounded Interpretations

arXiv:2507. 09751v3 Announce Type: replace Abstract: Large language models (LLMs) have demonstrated impressive capabilities in natural language understanding and generation, but exhibit problems with logical consistency in their output.

By Bradley P. Allen, Prateek Chhikara, Thomas Macaulay Ferguson, Filip Ilievski, Paul Groth

llms rag benchmarks safety

arXiv Machine Learning

Jun 9

Do Larger Models Really Win in Drug Discovery? A Benchmark Assessment of Model Scaling in AI-Driven Molecular Property and Activity Prediction

arXiv:2604. 26498v3 Announce Type: replace Abstract: The rapid growth of molecular foundation models and large language models (LLMs) has encouraged a scale centred view of AI in drug discovery, in which larger pretrained models are expected to supersede compact cheminformatics models.

By Jinjiang Guo, Sheng Ding

llms benchmarks safety

arXiv AI

Jun 9

Scaling Decision-Focused Learning to Large Problems with Lagrangian Decomposition

arXiv:2606. 08797v1 Announce Type: cross Abstract: Decision-focused learning has shown great promise for addressing predict-then-optimize problems, particularly in the presence of under-specified models.

By St\'ephane Eilles-Chan Way, Hugo Percot, Quentin Cappart, Tias Guns, Louis-Martin Rousseau

benchmarks

arXiv AI

Jun 9

Multi-Scale Feature Attention Network for Polymer Classification Using Terahertz Spectroscopy

arXiv:2606. 06554v2 Announce Type: replace-cross Abstract: Reliable polymer identification is essential for ensuring the quality and safety of recycled plastics, yet conventional sorting and spectroscopic techniques often struggle to deliver robust discrimination.

By Roshni Mahtani, Il\'an Carretero, Laura Monroy, Aldo Moreno-Oyervides, Oscar El\'ias Bonilla-Manrique, Roc\'io del Amor

benchmarks safety

arXiv AI

Jun 9

Data Synthesis and Parameter-Efficient Fine-Tuning for Low-Resource NMT: A Case Study on Q'eqchi' Mayan

arXiv:2606. 09767v1 Announce Type: cross Abstract: Neural machine translation for digitally low-resource Indigenous languages is often hindered by extreme data scarcity, prompting reliance on extractive web-scraping.

By Alexander Chulzhanov, Soeren Eberhardt, Arjun Mukherjee

llms nlp fine-tuning benchmarks

arXiv AI

Jun 9

A Systematic Study of Behavioral Cloning for Scientific Data Annotation

arXiv:2606. 07568v1 Announce Type: cross Abstract: Scientific data annotation, such as tracking animals in video or proofreading neural reconstructions, remains bottlenecked by the "last mile" problem: even with strong automation, verification and correction consume substantial human effort.

By Ishaan Singh Chandok, Core Francisco Park

fine-tuning benchmarks

arXiv Machine Learning

Jun 9

C$^3$ache: Accelerating World Action Models with Cross Inference Chunk Cache

arXiv:2606. 08962v1 Announce Type: new Abstract: World Action Models (WAMs) generalize better than standard Vision-Language-Action (VLA) policies to novel motions and environments, because a video-modeling objective lets them learn from abundant unlabeled video rather than scarce labeled robot demonstrations.

By Weisen Zhao, Lam Nguyen, Zhicong Lu, Yuzhang Shang

robotics multimodal benchmarks

arXiv AI

Jun 9

Polynomial Context-Truncation Sensitivity in Autoregressive Language Models: Sequential Wyner-Ziv Bounds for KV Cache Compression

arXiv:2605. 25085v2 Announce Type: replace-cross Abstract: We study the rate-distortion limits of online KV cache compression in autoregressive language models, formulating it as sequential Wyner-Ziv source coding on the filtration induced by the model, with the next-step query as decoder side information.

By Munsik Kim

llms efficiency benchmarks

Get the daily briefing