Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

24,507 stories · RSS feed

arXiv AI

Jun 2

Prune-OPD: Efficient and Reliable On-Policy Distillation for Long-Horizon Reasoning

arXiv:2605. 07804v3 Announce Type: replace-cross Abstract: On-policy distillation (OPD) leverages dense teacher rewards to enhance reasoning models.

By Zhicheng Yang, Zhijiang Guo, Yifan Song, Minrui Xu, Yongxin Wang, Yiwei Wang, Xiaodan Liang, Jing Tang

efficiency benchmarks

arXiv AI

Jun 2

Gradients with Respect to Semantics Preserving Embeddings Tell the Uncertainty of Large Language Models

arXiv:2605. 04638v2 Announce Type: replace-cross Abstract: Uncertainty quantification (UQ) is an important technique for ensuring the trustworthiness of LLMs, given their tendency to hallucinate.

By Mingda Li, Rundong Lv, Xinyu Li, Weinan Zhang, Ting Liu

llms rag benchmarks

arXiv AI

Jun 2

MedSynapse-V: Bridging Visual Perception and Clinical Intuition via Latent Memory Evolution

arXiv:2604. 26283v3 Announce Type: replace-cross Abstract: High-precision medical diagnosis relies not only on static imaging features but also on the implicit diagnostic memory experts instantly invoke during image interpretation.

By Chunzheng Zhu, Jiaqi Zeng, Junyu Jiang, Jianxin Lin, Yijun Wang

agents nlp reinforcement-learning efficiency benchmarks safety

arXiv AI

Jun 2

Frequency-Enhanced Diffusion Models: Curriculum-Guided Semantic Alignment for Zero-Shot Skeleton Action Recognition

arXiv:2604. 09063v3 Announce Type: replace-cross Abstract: Human action recognition is pivotal in computer vision, with applications ranging from surveillance to human-robot interaction.

By Yuxi Zhou, Zhengbo Zhang, Jingyu Pan, Zhiyu Lin, Zhigang Tu

diffusion computer-vision robotics benchmarks safety

arXiv AI

Jun 2

Claudini: Autoresearch Discovers State-of-the-Art Adversarial Attack Algorithms for LLMs

arXiv:2603. 24511v2 Announce Type: replace-cross Abstract: We show that AI agents are capable of discovering novel algorithms for adversarial attacks against LLMs, advancing the state of the art on white-box jailbreaking and prompt injection evaluations.

By Alexander Panfilov, Peter Romov, Igor Shilov, Yves-Alexandre de Montjoye, Jonas Geiping, Maksym Andriushchenko

llms agents multimodal benchmarks safety

arXiv AI

Jun 2

EuraGovExam: A Multilingual Multimodal Benchmark from Real-World Civil Service Exams

arXiv:2603. 27223v2 Announce Type: replace-cross Abstract: We present EuraGovExam, a multilingual and multimodal benchmark sourced from real-world civil service examinations across five representative Eurasian regions: South Korea, Japan, Taiwan, India, and the European Union.

By Jaeseong Kim, Chaehwan Lim, Sang Hyun Gil, Suan Lee

llms multimodal benchmarks

arXiv AI

Jun 2

Beyond String Matching: Semantic Evaluation of PDF Table Extraction

arXiv:2603. 18652v2 Announce Type: replace-cross Abstract: Reliably extracting tables from PDFs is essential for large-scale scientific data mining and knowledge base construction, yet existing evaluation approaches rely on rule-based metrics that fail to capture semantic equivalence of table content.

By Pius Horn, Janis Keuper

llms rag benchmarks

arXiv AI

Jun 2

Ethical Fairness in Ubiquitous Health Sensing without Known Attributes

arXiv:2603. 13373v4 Announce Type: replace-cross Abstract: In ubiquitous and mobile health systems, computational models infer human states from wearable, behavioral, and physiological sensing data.

By Shaily Roy, Harshit Sharma, Daniel A. Adler, Srijan Sen, Tanzeem Choudhury, Asif Salekin

benchmarks safety

arXiv Machine Learning

Jun 2

KISS: Keeping it Simple and Slotted when Learning to Communicate over Wireless

arXiv:2606. 00266v1 Announce Type: cross Abstract: A long-standing challenge in distributed wireless systems is ensuring efficient and fair random channel access.

By Kamil Szczech, Maksymilian Wojnar, Krzysztof Rusek, Katarzyna Kosek-Szott, Szymon Szott

agents benchmarks safety

arXiv Machine Learning

Jun 2

Autopilot-Preserving Residual Q-Learning with HJB-Inspired Finite-Action Risk Filtering for Fixed-Wing UAV Command Supervision

arXiv:2606. 01397v1 Announce Type: cross Abstract: A fixed-wing UAV must hold airspeed, altitude, and heading references under wind, gusts, and turbulence, channels coupled so that correcting one can degrade another.

By Mehmet Iscan, Batuhan Temiz

reinforcement-learning benchmarks

arXiv AI

Jun 2

DyLLM: Efficient Diffusion LLM Inference via Saliency-based Token Selection and Partial Attention

arXiv:2603. 08026v2 Announce Type: replace-cross Abstract: Masked diffusion language models enable parallel token decoding, providing a promising alternative to the sequential nature of autoregressive generation.

By Younjoo Lee, Seungkyun Dan, Junghoo Lee, Jaiyoung Park, Jung Ho Ahn

llms diffusion efficiency benchmarks

arXiv AI

Jun 2

Interpretable Multimodal Gesture Recognition for Drone and Mobile Robot Teleoperation via Log-Likelihood Ratio Fusion

arXiv:2602. 23694v3 Announce Type: replace-cross Abstract: Human operators are still frequently exposed to hazardous environments such as disaster zones and industrial facilities, where intuitive and reliable teleoperation of mobile robots and Unmanned Aerial Vehicles (UAVs) is essential.

By Seungyeol Baek, Jaspreet Singh, Lala Shakti Swarup Ray, Hymalai Bello, Paul Lukowicz, Sungho Suh

robotics multimodal benchmarks safety

arXiv AI

Jun 2

Are LLMs Ready for Neural-integrated Mechanistic Modeling? A Benchmark and Agentic Framework

arXiv:2602. 18008v2 Announce Type: replace-cross Abstract: Large language models (LLMs) have shown promise in constructing mechanistic models from data.

By Zihan Guan, Rituparna Datta, Mengxuan Hu, Shunshun Liu, Aiying Zhang, Prasanna Balachandran, Sheng Li, Anil Vullikanti

llms agents benchmarks

arXiv AI

Jun 2

When and How Much to Imagine: Adaptive Test-Time Scaling with World Models for Visual Spatial Reasoning

arXiv:2602. 08236v2 Announce Type: replace-cross Abstract: Despite rapid progress in MLLMs, visual spatial reasoning remains unreliable when correct answers depend on how a scene would appear under unseen or alternative viewpoints.

By Shoubin Yu, Yue Zhang, Zun Wang, Jaehong Yoon, Huaxiu Yao, Mingyu Ding, Mohit Bansal

llms robotics benchmarks

arXiv AI

Jun 2

Rethinking Scientific Modeling: Toward Physically Consistent and Simulation-Executable Programmatic Generation

arXiv:2602. 07083v2 Announce Type: replace-cross Abstract: Structural modeling is a fundamental component of computational engineering science, in which even minor physical inconsistencies or specification violations may invalidate downstream simulations.

By Yongqing Jiang, Jianze Wang, Zhiqi Shen, Zhenghong Lin, Jiayuan Wang, Yijian Yang, Kaoshan Dai, Haoran Luo

llms fine-tuning benchmarks safety

arXiv AI

Jun 2

From Evaluation to Design: Using Potential Energy Surface Smoothness Metrics to Guide Machine Learning Interatomic Potential Architectures

arXiv:2602. 04861v2 Announce Type: replace-cross Abstract: Machine Learning Interatomic Potentials (MLIPs) sometimes fail to reproduce the physical smoothness of the quantum potential energy surface (PES), leading to erroneous behavior in downstream simulations that standard energy and force regression evaluations can miss.

By Ryan Liu, Eric Qu, Tobias Kreiman, Samuel M. Blau, Aditi S. Krishnapriyan

llms benchmarks

arXiv AI

Jun 2

ASKD-Whisper: Adaptive Self-knowledge Distillation for Efficient and Low-Latency Automatic Speech Recognition

arXiv:2601. 19919v2 Announce Type: replace-cross Abstract: Knowledge distillation (KD) is one of the most effective paradigms for compressing large-scale foundation models into deployable architectures.

By Junseok Lee, Nahun Kim, Sangyong Lee, Chang-Jae Chun

efficiency multimodal benchmarks safety

arXiv AI

Jun 2

One Bias After Another: Mechanistic Reward Shaping and Persistent Biases in Language Reward Models

arXiv:2603. 03291v2 Announce Type: replace-cross Abstract: Reward Models (RMs) are crucial for online alignment of language models (LMs) with human preferences.

By Daniel Fein, Max Lamparth, Violet Xiang, Mykel J. Kochenderfer, Nick Haber

llms benchmarks safety

arXiv AI

Jun 2

VLM4VLA: Revisiting Vision-Language-Models in Vision-Language-Action Models

arXiv:2601. 03309v2 Announce Type: replace-cross Abstract: Vision-Language-Action (VLA) models, which integrate pretrained large Vision-Language Models (VLM) into their policy backbone, are gaining significant attention for their promising generalization capabilities.

By Jianke Zhang, Xiaoyu Chen, Qiuyue Wang, Mingsheng Li, Yanjiang Guo, Yucheng Hu, Jiajun Zhang, Shuai Bai, Junyang Lin, Jianyu Chen

llms computer-vision robotics fine-tuning multimodal benchmarks

arXiv AI

Jun 2

Prototypicality Bias Reveals Blindspots in Multimodal Evaluation Metrics

arXiv:2601. 04946v3 Announce Type: replace-cross Abstract: Automatic metrics are widely used to evaluate text-to-image models, often replacing human judgment in benchmarking, model selection, and large-scale data filtering.

By Subhadeep Roy, Gagan Bhatia, Steffen Eger

rag diffusion multimodal benchmarks safety

Get the daily briefing