Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

24,507 stories · RSS feed

arXiv AI

Jun 2

One Bias After Another: Mechanistic Reward Shaping and Persistent Biases in Language Reward Models

arXiv:2603. 03291v2 Announce Type: replace-cross Abstract: Reward Models (RMs) are crucial for online alignment of language models (LMs) with human preferences.

By Daniel Fein, Max Lamparth, Violet Xiang, Mykel J. Kochenderfer, Nick Haber

llms benchmarks safety

arXiv AI

Jun 2

BenHalluEval: A Multi-Task Hallucination Evaluation Framework for Large Language Models on Bengali

arXiv:2605. 31483v1 Announce Type: cross Abstract: Despite Bengali being the sixth most spoken language in the world, no prior work has systematically evaluated hallucination in large language models (LLMs) for Bengali.

By Shefayat E Shams Adib, Ahmed Alfey Sani, Ekramul Alam Esham, Ajwad Abrar, Ishmam Tashdeed, Md Taukir Azam Chowdhury

llms nlp benchmarks safety

arXiv AI

Jun 2

ASKD-Whisper: Adaptive Self-knowledge Distillation for Efficient and Low-Latency Automatic Speech Recognition

arXiv:2601. 19919v2 Announce Type: replace-cross Abstract: Knowledge distillation (KD) is one of the most effective paradigms for compressing large-scale foundation models into deployable architectures.

By Junseok Lee, Nahun Kim, Sangyong Lee, Chang-Jae Chun

efficiency multimodal benchmarks safety

arXiv AI

Jun 2

Token Predictors Are Not Planners: Building Physically Grounded Causal Reasoners

arXiv:2606. 01810v1 Announce Type: new Abstract: Current benchmarks for embodied vision-language planning often favor linguistic next-token prediction over physically grounded next-state reasoning.

By Zheng Lu, Mingqi Gao, Qinlei Xie, Wanqi Zhong, Hanwen Cui, Heng Cao, Zirui Song, Yifan Yang, Chong Luo, Bei Liu, Yiming Li

llms agents robotics multimodal benchmarks

arXiv AI

Jun 2

Rethinking Scientific Modeling: Toward Physically Consistent and Simulation-Executable Programmatic Generation

arXiv:2602. 07083v2 Announce Type: replace-cross Abstract: Structural modeling is a fundamental component of computational engineering science, in which even minor physical inconsistencies or specification violations may invalidate downstream simulations.

By Yongqing Jiang, Jianze Wang, Zhiqi Shen, Zhenghong Lin, Jiayuan Wang, Yijian Yang, Kaoshan Dai, Haoran Luo

llms fine-tuning benchmarks safety

arXiv AI

Jun 2

When and How Much to Imagine: Adaptive Test-Time Scaling with World Models for Visual Spatial Reasoning

arXiv:2602. 08236v2 Announce Type: replace-cross Abstract: Despite rapid progress in MLLMs, visual spatial reasoning remains unreliable when correct answers depend on how a scene would appear under unseen or alternative viewpoints.

By Shoubin Yu, Yue Zhang, Zun Wang, Jaehong Yoon, Huaxiu Yao, Mingyu Ding, Mohit Bansal

llms robotics benchmarks

arXiv AI

Jun 2

Demystifying Multi-Agent Debate: The Role of Confidence and Diversity

arXiv:2601. 19921v2 Announce Type: replace-cross Abstract: Multi-agent debate (MAD) is widely used to improve large language model (LLM) performance through test-time scaling, yet recent work shows that vanilla MAD often underperforms simple majority vote despite higher computational cost.

By Xiaochen Zhu, Caiqi Zhang, Yizhou Chi, Tom Stafford, Nigel Collier, Andreas Vlachos

llms agents benchmarks

arXiv AI

Jun 2

Are LLMs Ready for Neural-integrated Mechanistic Modeling? A Benchmark and Agentic Framework

arXiv:2602. 18008v2 Announce Type: replace-cross Abstract: Large language models (LLMs) have shown promise in constructing mechanistic models from data.

By Zihan Guan, Rituparna Datta, Mengxuan Hu, Shunshun Liu, Aiying Zhang, Prasanna Balachandran, Sheng Li, Anil Vullikanti

llms agents benchmarks

arXiv AI

Jun 2

Interpretable Multimodal Gesture Recognition for Drone and Mobile Robot Teleoperation via Log-Likelihood Ratio Fusion

arXiv:2602. 23694v3 Announce Type: replace-cross Abstract: Human operators are still frequently exposed to hazardous environments such as disaster zones and industrial facilities, where intuitive and reliable teleoperation of mobile robots and Unmanned Aerial Vehicles (UAVs) is essential.

By Seungyeol Baek, Jaspreet Singh, Lala Shakti Swarup Ray, Hymalai Bello, Paul Lukowicz, Sungho Suh

robotics multimodal benchmarks safety

arXiv AI

Jun 2

DyLLM: Efficient Diffusion LLM Inference via Saliency-based Token Selection and Partial Attention

arXiv:2603. 08026v2 Announce Type: replace-cross Abstract: Masked diffusion language models enable parallel token decoding, providing a promising alternative to the sequential nature of autoregressive generation.

By Younjoo Lee, Seungkyun Dan, Junghoo Lee, Jaiyoung Park, Jung Ho Ahn

llms diffusion efficiency benchmarks

arXiv AI

Jun 2

Ethical Fairness in Ubiquitous Health Sensing without Known Attributes

arXiv:2603. 13373v4 Announce Type: replace-cross Abstract: In ubiquitous and mobile health systems, computational models infer human states from wearable, behavioral, and physiological sensing data.

By Shaily Roy, Harshit Sharma, Daniel A. Adler, Srijan Sen, Tanzeem Choudhury, Asif Salekin

benchmarks safety

arXiv AI

Jun 2

Beyond String Matching: Semantic Evaluation of PDF Table Extraction

arXiv:2603. 18652v2 Announce Type: replace-cross Abstract: Reliably extracting tables from PDFs is essential for large-scale scientific data mining and knowledge base construction, yet existing evaluation approaches rely on rule-based metrics that fail to capture semantic equivalence of table content.

By Pius Horn, Janis Keuper

llms rag benchmarks

arXiv AI

Jun 2

EuraGovExam: A Multilingual Multimodal Benchmark from Real-World Civil Service Exams

arXiv:2603. 27223v2 Announce Type: replace-cross Abstract: We present EuraGovExam, a multilingual and multimodal benchmark sourced from real-world civil service examinations across five representative Eurasian regions: South Korea, Japan, Taiwan, India, and the European Union.

By Jaeseong Kim, Chaehwan Lim, Sang Hyun Gil, Suan Lee

llms multimodal benchmarks

arXiv AI

Jun 2

Claudini: Autoresearch Discovers State-of-the-Art Adversarial Attack Algorithms for LLMs

arXiv:2603. 24511v2 Announce Type: replace-cross Abstract: We show that AI agents are capable of discovering novel algorithms for adversarial attacks against LLMs, advancing the state of the art on white-box jailbreaking and prompt injection evaluations.

By Alexander Panfilov, Peter Romov, Igor Shilov, Yves-Alexandre de Montjoye, Jonas Geiping, Maksym Andriushchenko

llms agents multimodal benchmarks safety

arXiv AI

Jun 2

Frequency-Enhanced Diffusion Models: Curriculum-Guided Semantic Alignment for Zero-Shot Skeleton Action Recognition

arXiv:2604. 09063v3 Announce Type: replace-cross Abstract: Human action recognition is pivotal in computer vision, with applications ranging from surveillance to human-robot interaction.

By Yuxi Zhou, Zhengbo Zhang, Jingyu Pan, Zhiyu Lin, Zhigang Tu

diffusion computer-vision robotics benchmarks safety

arXiv AI

Jun 2

MedSynapse-V: Bridging Visual Perception and Clinical Intuition via Latent Memory Evolution

arXiv:2604. 26283v3 Announce Type: replace-cross Abstract: High-precision medical diagnosis relies not only on static imaging features but also on the implicit diagnostic memory experts instantly invoke during image interpretation.

By Chunzheng Zhu, Jiaqi Zeng, Junyu Jiang, Jianxin Lin, Yijun Wang

agents nlp reinforcement-learning efficiency benchmarks safety

arXiv AI

Jun 2

Gradients with Respect to Semantics Preserving Embeddings Tell the Uncertainty of Large Language Models

arXiv:2605. 04638v2 Announce Type: replace-cross Abstract: Uncertainty quantification (UQ) is an important technique for ensuring the trustworthiness of LLMs, given their tendency to hallucinate.

By Mingda Li, Rundong Lv, Xinyu Li, Weinan Zhang, Ting Liu

llms rag benchmarks

arXiv AI

Jun 2

Prune-OPD: Efficient and Reliable On-Policy Distillation for Long-Horizon Reasoning

arXiv:2605. 07804v3 Announce Type: replace-cross Abstract: On-policy distillation (OPD) leverages dense teacher rewards to enhance reasoning models.

By Zhicheng Yang, Zhijiang Guo, Yifan Song, Minrui Xu, Yongxin Wang, Yiwei Wang, Xiaodan Liang, Jing Tang

efficiency benchmarks

arXiv AI

Jun 2

Multi-Rollout On-Policy Distillation via Peer Successes and Failures

arXiv:2605. 12652v2 Announce Type: replace-cross Abstract: Large language models are often post-trained with sparse verifier rewards, which indicate whether a sampled trajectory succeeds but provide limited guidance about where reasoning succeeds or fails.

By Weichen Yu, Xiaomin Li, Yizhou Zhao, Xiaoze Liu, Ruowang Zhang, Haixin Wang, Yinyi Luo, Chen Henry Wu, Gaurav Mittal, Matt Fredrikson, Yu Hu

llms nlp efficiency benchmarks

arXiv AI

Jun 2

Avoiding Structural Failure Modes in Tabular Fair SSL: Online Primal-Dual Allocation under Confidence Gating

arXiv:2605. 16446v2 Announce Type: replace-cross Abstract: Semi-supervised learning (SSL) enables prediction with limited labels, but high-stakes tabular applications (medical, credit, recidivism) require statistical fairness guarantees.

By Hangchuan Liang, Changchun Li

benchmarks safety

Get the daily briefing