Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

24,805 stories · RSS feed

arXiv AI

Jun 2

VLM4VLA: Revisiting Vision-Language-Models in Vision-Language-Action Models

arXiv:2601. 03309v2 Announce Type: replace-cross Abstract: Vision-Language-Action (VLA) models, which integrate pretrained large Vision-Language Models (VLM) into their policy backbone, are gaining significant attention for their promising generalization capabilities.

By Jianke Zhang, Xiaoyu Chen, Qiuyue Wang, Mingsheng Li, Yanjiang Guo, Yucheng Hu, Jiajun Zhang, Shuai Bai, Junyang Lin, Jianyu Chen

llms computer-vision robotics fine-tuning multimodal benchmarks

arXiv AI

Jun 2

One Bias After Another: Mechanistic Reward Shaping and Persistent Biases in Language Reward Models

arXiv:2603. 03291v2 Announce Type: replace-cross Abstract: Reward Models (RMs) are crucial for online alignment of language models (LMs) with human preferences.

By Daniel Fein, Max Lamparth, Violet Xiang, Mykel J. Kochenderfer, Nick Haber

llms benchmarks safety

arXiv AI

Jun 2

ASKD-Whisper: Adaptive Self-knowledge Distillation for Efficient and Low-Latency Automatic Speech Recognition

arXiv:2601. 19919v2 Announce Type: replace-cross Abstract: Knowledge distillation (KD) is one of the most effective paradigms for compressing large-scale foundation models into deployable architectures.

By Junseok Lee, Nahun Kim, Sangyong Lee, Chang-Jae Chun

efficiency multimodal benchmarks safety

arXiv AI

Jun 2

Silent Failures in Physical AI: A Literature Review of Runtime Action Authorization for Autonomous Systems

arXiv:2606. 00090v1 Announce Type: cross Abstract: Physical AI systems increasingly map multimodal observations, language instructions, and learned world representations into physically consequential actions.

By Barak Or

agents robotics multimodal benchmarks safety

arXiv AI

Jun 2

TrustLDM: Benchmarking Trustworthiness in Language Diffusion Models

arXiv:2606. 00023v1 Announce Type: cross Abstract: The rapid development of Language Diffusion Models (LDMs) challenges the dominant position of auto-regressive competitors in language processing.

By Yichuan Mo, Yukun Jiang, Yanbo Shi, Mingjie Li, Michael Backes, Yang Zhang, Yisen Wang

diffusion benchmarks safety

arXiv AI

Jun 2

V2I Work Zone Geometry Reconstruction with Pose-Conditioned UWB Range Denoising

arXiv:2606. 00119v1 Announce Type: cross Abstract: Reliable work zone mapping is important for connected and autonomous vehicles (CAVs) to navigate safely and smoothly through work zone areas.

By Jiaxi Liu, Hangyu Li, Yang Cheng, Rui Gana, Junwei You, Weizhe Tang, Peng Zhang, Steven T. Parker, Xiaopeng Li, Bin Ran

agents benchmarks

arXiv AI

Jun 2

Community-Aware Assessment of Social Textual Engagement and Resonance: A Human-Centric Perspective on User-Generated Content Evaluation

arXiv:2606. 01897v1 Announce Type: new Abstract: Traditional Video Quality Assessment (VQA) focuses narrowly on aesthetic fidelity, overlooking the complex social dynamics that define quality in User-Generated Content (UGC).

By Tianjiao Li, Kai Zhao, Xiang Li, Yang Liu, Huyang Sun

reinforcement-learning fine-tuning multimodal benchmarks safety

arXiv AI

Jun 2

On the Difficulty of Learning a Meta-network for Training Data Selection

arXiv:2606. 00571v1 Announce Type: cross Abstract: Synthetic data are increasingly used to train neural networks, yet distributional mismatch with real data limits their effectiveness when used indiscriminately.

By Zilin Du, Junqi Zhao, Boyang Albert Li

benchmarks data-engineering

arXiv AI

Jun 2

From Evaluation to Design: Using Potential Energy Surface Smoothness Metrics to Guide Machine Learning Interatomic Potential Architectures

arXiv:2602. 04861v2 Announce Type: replace-cross Abstract: Machine Learning Interatomic Potentials (MLIPs) sometimes fail to reproduce the physical smoothness of the quantum potential energy surface (PES), leading to erroneous behavior in downstream simulations that standard energy and force regression evaluations can miss.

By Ryan Liu, Eric Qu, Tobias Kreiman, Samuel M. Blau, Aditi S. Krishnapriyan

llms benchmarks

arXiv AI

Jun 2

MLLM-Microscope: Unlocking Hidden Structure Within Multimodal Large Language Models

arXiv:2606. 00909v1 Announce Type: cross Abstract: This work presents MLLM-Microscope, a novel system designed for analyzing the hidden representations within Multimodal Large Language Models (MLLMs).

By Ravil Mussabayev, Rustam Mussabayev

llms rag multimodal benchmarks

arXiv AI

Jun 2

CV-Arena: An Open Benchmark for Instructional Computer Vision Problem Solving with Human-AI Collaborative Preferences

arXiv:2606. 00931v1 Announce Type: cross Abstract: Instruction-guided image editing is becoming a general interface for visual work, yet existing benchmarks still focus largely on narrow appearance edits and do not fully capture the diversity of real-image tasks in professional workflows.

By Fangzhou Lin, Peiran Li, Lingyu Xu, Wenjing Chen, Qianwen Ge, Shuo Xing, Mingyang Wu, Xiangbo Gao, Siyuan Yang, Kazunori Yamada, Ziming Zhang, Haichong Zhang, Zhen Dong, Ming-Hsuan Yang, Zhengzhong Tu

agents computer-vision multimodal benchmarks

arXiv AI

Jun 2

Accuracy, Stability, and Repeated-Run Reliability of Large Language Models on Deterministic Programming Tasks

arXiv:2606. 00920v1 Announce Type: cross Abstract: Run-level pass rate overstates retry-free coverage by up to 17.

By Yongxi Zhou, Lai Yun Choi, Jiaxi Wen, Wenbo Ye

llms benchmarks

arXiv AI

Jun 2

TimeSage-MT: A Multi-Turn Benchmark for Evaluating Agentic Time Series Reasoning

arXiv:2606. 01498v1 Announce Type: cross Abstract: Time series data inform critical decisions across many real-world domains.

By Yaxuan Kong, Qingren Yao, Yuqi Nie, Yichen Li, Yilei Shao, Stefan Zohren, Anna Vettoruzzo, Joaquin Vanschoren, Ming Jin, Qingsong Wen

llms agents benchmarks

arXiv Machine Learning

Jun 2

EEG-FuseFormer: A Transformer-Driven Feature Fusion Framework for Seizure Onset Prediction

arXiv:2606. 02166v1 Announce Type: new Abstract: Epilepsy is one of the most common neurological disorders globally, characterized by recurring seizures and significantly impacting the quality of life.

By Vigneshwar Hariharan (National University of Singapore), Chithra Reghuvaran (University College Dublin), Arlene John (University of Twente), Nhat Pham (Cardiff University), Omer Rana (Cardiff University), Deepu John (University College Dublin), Ganesh Neelakanta Iyer (National University of Singapore)

llms fine-tuning benchmarks

arXiv AI

Jun 2

Rethinking Scientific Modeling: Toward Physically Consistent and Simulation-Executable Programmatic Generation

arXiv:2602. 07083v2 Announce Type: replace-cross Abstract: Structural modeling is a fundamental component of computational engineering science, in which even minor physical inconsistencies or specification violations may invalidate downstream simulations.

By Yongqing Jiang, Jianze Wang, Zhiqi Shen, Zhenghong Lin, Jiayuan Wang, Yijian Yang, Kaoshan Dai, Haoran Luo

llms fine-tuning benchmarks safety

arXiv AI

Jun 2

DAStatFormer: A Hybrid Multibranch Transformer with Statistical Feature Integration for DAS-Based Pattern Recognitions

arXiv:2606. 00081v1 Announce Type: cross Abstract: Distributed Acoustic Sensing (DAS) enables large-scale monitoring through optical fibers, but its high dimensionality and complex spatio-temporal patterns make event classification demanding.

By Michel Dione (CERI SN - IMT Nord Europe), Jerry Lonlac (CERI SN - IMT Nord Europe), H\'el\`ene Louis (CERI SN - IMT Nord Europe), Anthony Fleury (CERI SN - IMT Nord Europe), Stephane Lecoeuche

llms benchmarks

arXiv AI

Jun 2

When and How Much to Imagine: Adaptive Test-Time Scaling with World Models for Visual Spatial Reasoning

arXiv:2602. 08236v2 Announce Type: replace-cross Abstract: Despite rapid progress in MLLMs, visual spatial reasoning remains unreliable when correct answers depend on how a scene would appear under unseen or alternative viewpoints.

By Shoubin Yu, Yue Zhang, Zun Wang, Jaehong Yoon, Huaxiu Yao, Mingyu Ding, Mohit Bansal

llms robotics benchmarks

arXiv AI

Jun 2

Safety Mirage: How Spurious Correlations Undermine VLM Safety Fine-Tuning and Can Be Mitigated by Machine Unlearning

arXiv:2503. 11832v5 Announce Type: replace Abstract: Recent vision language models (VLMs) have made remarkable strides in generative modeling with multimodal inputs, particularly text and images.

By Yiwei Chen, Yuguang Yao, Yihua Zhang, Bingquan Shen, Gaowen Liu, Sijia Liu

llms fine-tuning multimodal benchmarks safety

arXiv AI

Jun 2

Why Not Hyperparameter-Friendly Optimisation? A Monotonic Adaptive Norm Rescaling Approach For Long-Tailed Recognition

arXiv:2606. 02526v1 Announce Type: cross Abstract: Long-tailed recognition poses a significant challenge for deep learning.

By Shuo Zhang, Chenqi Li, Tingting Zhu

benchmarks

arXiv AI

Jun 2

AdaCodec: A Predictive Visual Code for Video MLLMs

arXiv:2606. 02569v1 Announce Type: cross Abstract: Video is temporally redundant: adjacent frames usually share most objects, background, and layout.

By Haowen Hou, Zhen Huang, Zheming Liang, Qingyi Si, Chenglin Li, Shuai Dong, Kele Shao, Ruilin Li, Dianyi Wang, Nan Duan, Jiaqi Wang

llms multimodal benchmarks

Get the daily briefing