Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

22,245 stories · RSS feed

arXiv Machine Learning

Jun 9

Cutting LLM Evaluation Costs with SySRs: A Bandit Algorithm that Provably Exploits Model Similarity

arXiv:2606. 07726v1 Announce Type: new Abstract: Large Language Models are typically benchmarked by evaluating every model on every test query.

By Zifan Lyu, Chahine Nejma, Tobias Wegel, Fanny Yang, Florian E. Dorner

llms reinforcement-learning benchmarks

arXiv Machine Learning

Jun 9

UNIQ: Conformal Calibration for Adaptive Conservatism in Offline Reinforcement Learning

arXiv:2606. 07592v1 Announce Type: new Abstract: Offline reinforcement learning requires careful conservatism to mitigate distribution shift, yet most existing methods apply a fixed penalty uniformly across all states regardless of local data coverage.

By Aditya Upadhyay

reinforcement-learning benchmarks

arXiv AI

Jun 9

Structure-Conditioned Actor-Critic Branches for Quality-Diversity Reinforcement Learning

arXiv:2606. 08735v1 Announce Type: new Abstract: Quality-diversity reinforcement learning (QD-RL) aims to construct policy repertoires that contain both high-performing and behaviorally diverse policies.

By Lianrong Zuo, Peilan Xu, Yong Liu, Wenjian Luo

reinforcement-learning efficiency benchmarks

arXiv AI

Jun 9

TheoremBench: Evaluating LLMs on Theorem Proving in Formal Mathematics

arXiv:2606. 09450v1 Announce Type: new Abstract: LLMs have recently achieved strong results on formal proving benchmarks.

By QuocViet Pham, Elvir Karimov, Andrey Galichin, Ivan Oseledets

llms benchmarks

arXiv AI

Jun 9

RiskNet: A large-scale dataset of AI risk incidents from news with alignment and multi-dimensional annotations

arXiv:2606. 08376v1 Announce Type: cross Abstract: As artificial intelligence (AI) systems are increasingly deployed across socially consequential domains, reports of AI-related harms and failures have grown in frequency and diversity.

By Leihan Zhang, Wecheng Ye, Xianlong Ma, Haochuan Liu, Yang Li, Qianyu Zhang, Jinliang Chen, Qiang Yan

benchmarks safety

arXiv Machine Learning

Jun 9

When Do Local Score Models Extrapolate Across Size? A Diagnostic Theory and Benchmark

arXiv:2606. 09705v1 Announce Type: new Abstract: Scientific generative modeling often requires size transfer, where models trained on small systems are evaluated on larger ones.

By Wenjie Xi

diffusion benchmarks

arXiv AI

Jun 9

Correlation Is Not Enough: Embedding Human Metadata for Individual Causal Discovery

arXiv:2606. 09672v1 Announce Type: new Abstract: Ask a pretrained biomedical language model whether "cortisol 28 ug/dL" and "stock-market volatility" are related, and it returns a cosine similarity of 0.

By Suraj Biswas, Saurabh Gupta, Pritam Mukherjee

llms rag benchmarks

arXiv AI

Jun 9

From USD Scenes to Knowledge Graphs: Zero-Shot Ontology Grounding with LLMs

arXiv:2606. 09134v1 Announce Type: cross Abstract: Constructing knowledge graphs from 3D simulation scenes is essential for robot task reasoning, but the key bottleneck, grounding scene objects to formal ontology classes, still relies on manually curated dictionaries that are brittle and do not generalize across assets.

By Jiangtao Shuai, Zongxiong Chen, Manfred Hauswirth, Sonja Schimmler

llms rag robotics benchmarks

arXiv AI

Jun 9

SecureVibeBench: Benchmarking Secure Vibe Coding of AI Agents via Reconstructing Vulnerability-Introducing Scenarios

arXiv:2509. 22097v5 Announce Type: replace-cross Abstract: Large language model-powered code agents are rapidly transforming software engineering, yet the security risks of their generated code have become a critical concern.

By Junkai Chen, Huihui Huang, Yunbo Lyu, Junwen An, Jieke Shi, Chengran Yang, Ting Zhang, Haoye Tian, Yikun Li, Zhenhao Li, Xin Zhou, Xing Hu, David Lo

llms agents benchmarks

arXiv AI

Jun 9

Riemannian-Manifold Steering: Geometry-Aware Generative Autoencoders for Label-Free Steering

arXiv:2605. 24942v2 Announce Type: replace-cross Abstract: Steering a language model - intervening on its internal activations to change downstream behaviour - has recently expanded beyond linear interpolation to nonlinear methods such as angular and kernelized steering, which define intervention transformations without learning an explicit geometry over paths in activation space.

By Narmeen Oozeer, Shivam Raval, Philip Quirke, Manikandan Ravikiran, Jeff Phillips, Shriyash Upadhyay, Amirali Abdullah

llms benchmarks

arXiv AI

Jun 9

InA-Probe: Instruction-Aware Active Probing for Time Series Forecasting with LLMs

arXiv:2606. 08601v1 Announce Type: new Abstract: Large Language Models (LLMs) have recently demonstrated impressive potential for time series forecasting.

By Peiliang Gong, Emadeldeen Eldele, Chenyu Liu, Ziyu Jia, Yi Ding, Xinliang Zhou, Lianchao Gu, Qi Zhu, Yang Liu, Daoqiang Zhang, Xiaoli Li

llms benchmarks safety

arXiv AI

Jun 9

ConflictRAG: Detecting and Resolving Knowledge Conflicts in Retrieval Augmented Generation

arXiv:2605. 17301v2 Announce Type: replace-cross Abstract: Retrieval-Augmented Generation (RAG) systems implicitly assume mutual consistency among retrieved documents -- an assumption that frequently fails in practice.

By Chenyu Wang, Yueyuan Li, Yingmin Liu, Yang Shu

llms rag benchmarks

arXiv AI

Jun 9

Beyond Pass Rate: A Multilingual, Execution-Grounded Evaluation of Open Code LLMs

arXiv:2606. 08840v1 Announce Type: new Abstract: Code generation models are typically compared using compact execution benchmarks and aggregate pass rates, but such summaries obscure how performance varies across programming languages, problem families, and failure modes.

By Sayed Erfan Arefin

llms benchmarks

arXiv AI

Jun 9

TQA-Bench: Evaluating LLMs for Multi-Table Question Answering

arXiv:2411. 19504v2 Announce Type: replace Abstract: The advance of large language models (LLMs) has unlocked great opportunities in complex multi-modal data management tasks, particularly in question answering (QA) over complicated multi-table relational data.

By Zipeng Qiu, Chenyue Li, You Peng, Guangxin He, Binhang Yuan, Chen Wang

llms nlp multimodal benchmarks

arXiv AI

Jun 9

HARBOR: A Harness Framework for Agentic Robot Reinforcement Learning

arXiv:2606. 08610v1 Announce Type: cross Abstract: Reinforcement learning (RL) has become a powerful paradigm for robot learning, particularly in sim-to-real settings, but its broader adoption remains limited by the engineering pipeline surrounding the algorithms.

By Zechu Li, Yufeng Jin, Xiaoyang Liu, Puze Liu, Vignesh Prasad, Carlo D'Eramo, Georgia Chalvatzaki

agents reinforcement-learning robotics benchmarks

arXiv AI

Jun 9

Cross-View Urban Traffic Dataset: Drone-Supervised Ground Truth for Monocular Bird's-Eye View Localization

arXiv:2606. 07708v1 Announce Type: cross Abstract: We introduce a dataset and benchmark for cross-view urban traffic perception built from synchronized ego-centric bicycle videos and aerial drone videos recorded at real urban intersections.

By Prakhar Bhardwaj, Simone Weikl, Kilian Mang, Elia Jonas Sandtner

robotics benchmarks safety

arXiv AI

Jun 9

More Yap Less Meaning: Uncovering Self-Improvement Behavior in SLMs

arXiv:2606. 08471v1 Announce Type: cross Abstract: Recently, language models have made rapid progress across various domains and applications.

By Marina Igitkhanian, Erik Arakelyan

llms benchmarks

arXiv AI

Jun 9

SpatialWorld: Benchmarking Interactive Spatial Reasoning of Multimodal Agents in Real-World Tasks

arXiv:2606. 09669v1 Announce Type: new Abstract: Spatial reasoning is a foundational capability for multimodal large language models (MLLMs) to perceive and operate within the physical world.

By Hongcheng Gao, Hailong Qu, Jingyi Tang, Jiahao Wang, Zihao Huang, Hengkang Qiao, Shihong Huang, Junming Yang, Yi Li, Hongyixuan Yuan, Wenjie Li, Bohan Zeng, Wenbo Li, Bo Wang, Jianhui Liu, Olive Huang, Haoyang Huang, Wentao Zhang, Guoqing Huang, Nan Duan, Yinpeng Dong

llms agents multimodal benchmarks

arXiv AI

Jun 9

GIScholarBench: Benchmarking LLM Overconfidence in GIS Research

arXiv:2606. 08036v1 Announce Type: cross Abstract: Large language models (LLMs) are increasingly used in academic research workflows, but scholarly tasks require high factual precision and therefore expose a key weakness: overconfidence.

By Zongrng Li, Mingzheng Yang, Lei Zou, Hongxu Ma, Hao Tian, Siqi Zhou, Wenjing Gong, Kaili Zhang, Bingqian Chen, Mitch Zhang, Yifan Yang

llms benchmarks

arXiv Machine Learning

Jun 9

Counterfactual Transport Flows for Offline Conservative Trajectory Refinement

arXiv:2606. 09115v1 Announce Type: new Abstract: Offline reinforcement learning (RL) offers a path to policy improvement from logged data alone, using historical returns or other measurable outcomes as world feedback.

By Lena Krieger, Xuan Zhao, Zhuo Cao, Qin Wang, Hanno Scharr, Ira Assent

reinforcement-learning benchmarks

Get the daily briefing