Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

24,805 stories · RSS feed

arXiv AI

Jun 2

Benchmarks for Vision-Language Models in Urban Perception Should Be Reliability-Aware and Negotiated

arXiv:2606. 00871v1 Announce Type: cross Abstract: Vision-language models (VLMs) are increasingly used to generate structured descriptions of street-level imagery for tasks such as streetscape auditing, mapping, and public consultation.

By Rashid Mushkani

llms multimodal benchmarks safety

arXiv AI

Jun 2

DetailMaster: Can Your Text-to-Image Model Handle Long Prompts?

arXiv:2505. 16915v3 Announce Type: replace-cross Abstract: While recent Text-to-Image (T2I) models show impressive capabilities in synthesizing images from brief descriptions, they struggle with the long, detailed prompts required for professional applications.

By Qirui Jiao, Daoyuan Chen, Yilun Huang, Xika Lin, Ying Shen, Yaliang Li

diffusion benchmarks

arXiv AI

Jun 2

SafeSteer: Localized On-Policy Distillation for Efficient Safety Alignment

arXiv:2606. 02530v1 Announce Type: new Abstract: Aligning Large Language Models (LLMs) with human values often degrades their general capabilities, termed the alignment tax.

By Hao Li, Jingkun An, Zijun Song, Pengyu Zhu, Rui Li, Hao Wang, Wendi Feng, Yesheng Liu, Lijun Li, Jin-Ge Yao, Lei Sha

llms efficiency benchmarks safety

arXiv AI

Jun 2

Taming System Complexity: Demystifying Software Engineering Agents in Diagnosing Linux Kernel Faults

arXiv:2505. 19489v2 Announce Type: replace Abstract: The Linux kernel is a critical system, serving as the foundation for numerous systems.

By Zhenhao Zhou, Zhuochen Huang, Yike He, Chong Wang, Jiajun Wang, Yijian Wu, Xin Peng, Yiling Lou

llms agents benchmarks

arXiv AI

Jun 2

Uncovering Competency Gaps in Large Language Models and Their Benchmarks

arXiv:2512. 20638v2 Announce Type: replace-cross Abstract: The evaluation of large language models relies heavily on standardized benchmarks.

By Maty Bohacek, Nino Scherrer, Nicholas Dufour, Thomas Leung, Christoph Bregler, Stephanie C. Y. Chan

llms benchmarks

arXiv AI

Jun 2

Skill-Based Mixture-of-Experts: Adaptive Routing for Heterogeneous Reasoning via Inferred Skills

arXiv:2503. 05641v4 Announce Type: replace-cross Abstract: Combining existing pre-trained LLMs is a promising approach for diverse reasoning tasks.

By Justin Chih-Yao Chen, Sukwon Yun, Elias Stengel-Eskin, Tianlong Chen, Mohit Bansal

llms agents benchmarks

arXiv AI

Jun 2

TCAR-Gen: Temporal Graph Retrieval with Evidence Fusion for Knowledge-Grounded Generation

arXiv:2606. 00029v1 Announce Type: cross Abstract: Retrieval-augmented generation systems struggle with temporal reasoning and evidence fusion when answering complex questions over historical criminal case narratives.

By Sidra Nasir, Muhammad Noman Zahid, Rizwan Ahmed Khan

llms rag nlp benchmarks

arXiv AI

Jun 2

Latent Reward Steering: An Adaptive Inference-Time Framework that Implicitly Promotes Cognitive Behaviors in Reasoning LLMs

arXiv:2606. 00726v1 Announce Type: new Abstract: Strong reasoning depends not only on model knowledge but also on how effectively cognitive behaviors are deployed during generation.

By Jiakang Li, Guanyu Zhu, Can Jin, Chenxi Huang, Dexu Yu, Ronghao Chen, Yang Zhou, Hongwu Peng, Xuanqi Lan, Dimitris N. Metaxas, Youhua Li

llms reinforcement-learning benchmarks

arXiv AI

Jun 2

The Cartesian Shortcut: Re-evaluate Vision Reasoning in Polar Coordinate Space

arXiv:2605. 09883v2 Announce Type: replace-cross Abstract: As current Multimodal Large Language Models rapidly saturate canonical visual reasoning benchmarks, a key question emerges: do these strong scores genuinely reflect robust visual understanding?

By Xia Hu, Zhenrui Yue, Brian Potetz, Howard Zhou, Leonidas Guibas, Chun-Ta Lu, Zhicheng Wang

llms multimodal benchmarks

arXiv AI

Jun 2

Task diversity produces systematic transfer but inhibits continual reinforcement learning

arXiv:2606. 00880v1 Announce Type: cross Abstract: Continual reinforcement learning aims to produce agents that learn not only to improve at their current tasks but also to adapt as task distributions change.

By Purab Seth, Neil Shah, Kunal Jha, Samuel J. Gershman, Max Kleiman-Weiner, Wilka Carvalho

agents reinforcement-learning benchmarks

arXiv AI

Jun 2

TukaBench: A Culturally Grounded Jailbreak Benchmark for African Languages

arXiv:2606. 01322v1 Announce Type: cross Abstract: Safety evaluation of Large Language Models (LLMs) remains heavily English-centric, leaving Low-Resource Languages (LRLs), particularly African ones, critically underexplored.

By Victor Akinode, Senyu Li, Wassim Hamidouche, Waqas Zamir, Inbal Becker-Reshef, David Ifeoluwa Adelani

llms benchmarks safety

arXiv Machine Learning

Jun 2

GRASP: Plan-Guided Graph Retrieval with Adaptive Fusion and Reranking on Semi-Structured Knowledge Bases

arXiv:2605. 30237v2 Announce Type: replace-cross Abstract: Semi-structured knowledge bases (SKBs) embed textual documents in a typed graph of entities and relations, and underpin applications such as product search, academic paper search, and precision-medicine inquiries.

By Yicheng Tao, Yiqun Wang, Xiangchen Song, Xin Luo, Kai Liu, Jie Liu

rag fine-tuning benchmarks

arXiv Machine Learning

Jun 2

Decision-Focused On-Policy Learning for Contextual Linear Optimization with Partial Feedback

arXiv:2606. 01081v1 Announce Type: new Abstract: Decision-focused learning (DFL) trains predictive models by optimizing downstream decision quality rather than standalone prediction accuracy.

By Wyame Benslimane, Tinghan Ye, Pascal Van Hentenryck, Paul Grigas

reinforcement-learning benchmarks

arXiv AI

Jun 2

StressDream: Steering Video World Models for Robust Policy Evaluation and Improvement

arXiv:2606. 00267v1 Announce Type: cross Abstract: Video world models (WMs) have shown promise for policy evaluation and improvement by imagining realistic future observations conditioned on ego-robot actions.

By Junwon Seo, Sushant Veer, Ran Tian, Wenhao Ding, Apoorva Sharma, Karen Leung, Edward Schmerling, Marco Pavone, Andrea Bajcsy

llms agents diffusion robotics multimodal benchmarks

arXiv Machine Learning

Jun 2

Echo State Networks for Time Series Forecasting: Hyperparameter Sweep and Benchmarking

arXiv:2602. 03912v4 Announce Type: replace Abstract: This paper investigates the performance of Echo State Networks (ESNs) for univariate forecasting of monthly and quarterly time series from the M4 Forecasting Competition dataset.

By Alexander H\"au{\ss}er

benchmarks

arXiv AI

Jun 2

Boosting Multimodal Federated Learning via Chained Modality Optimization

arXiv:2606. 01856v1 Announce Type: cross Abstract: Multimodal Federated Learning (MMFL) enables privacy-preserving collaborative learning across decentralized clients with heterogeneous data and modality availability.

By Zixin Zhang, Fan Qi, Shuai Li, Xiaoshan Yang, Changsheng Xu

multimodal benchmarks safety

arXiv AI

Jun 2

LookWise: Knowing When and Where to Look for Fine-Grained Visual Reasoning in Multimodal Large Language Models

arXiv:2603. 00171v3 Announce Type: replace-cross Abstract: Multimodal Large Language Models (MLLMs) are shifting towards "Thinking with Images" by actively exploring image details.

By Yuxiang Shen, Hailong Huang, Zhenkun Gao, Xueheng Li, Man Zhou, Chengjun Xie, Haoxuan Che, Xuanhua He, Jie Zhang

llms multimodal benchmarks

arXiv Machine Learning

Jun 2

Knowledge-Informed Kernel State Reconstruction from Heterogeneous Partial Observations

arXiv:2601. 22328v2 Announce Type: replace Abstract: Real-world scientific systems are rarely observed through complete, regularly sampled state trajectories.

By Luca Muscarnera, Silas Ruhrberg Est\'evez, Samuel Holt, Evgeny Saveliev, Mihaela van der Schaar

benchmarks

arXiv AI

Jun 2

When AI Benchmarks Plateau: A Systematic Study of Benchmark Saturation

arXiv:2602. 16763v2 Announce Type: replace Abstract: Artificial intelligence benchmarks are an important mechanism for measuring model progress and guiding deployment decisions.

By Mubashara Akhtar, Anka Reuel, Prajna Soni, Sanchit Ahuja, Pawan Sasanka Ammanamanchi, Ruchit Rawal, Vil\'em Zouhar, Srishti Yadav, Chenxi Whitehouse, Dayeon Ki, Jennifer Mickel, Leshem Choshen, Marek \v{S}uppa, Jan Batzner, Jenny Chim, Jeba Sania, Yanan Long, Hossein A. Rahmani, Christina Knight, Yiyang Nan, Jyoutir Raj, Yu Fan, Shubham Singh, Subramanyam Sahoo, Eliya Habba, Usman Gohar, Siddhesh Pawar, Robert Scholz, Arjun Subramonian, Jingwei Ni, Mykel Kochenderfer, Sanmi Koyejo, Mrinmaya Sachan, Stella Biderman, Zeerak Talat, Avijit Ghosh, Irene Solaiman

llms benchmarks

arXiv Machine Learning

Jun 2

From Zero to Hero: Advancing Zero-Shot Foundation Models for Tabular Outlier Detection

arXiv:2602. 03018v2 Announce Type: replace Abstract: Outlier detection (OD) is widely used in practice; but its effective deployment on new tasks is hindered by lack of labeled outliers, which makes algorithm and hyperparameter selection notoriously hard.

By Xueying Ding, Haomin Wen, Simon Kl\"uttermann, Leman Akoglu

llms benchmarks

Get the daily briefing