Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

24,805 stories · RSS feed

arXiv AI

Jun 2

PlanarBench: Evaluating LLM Spatial Reasoning via Planar Graph Drawing

arXiv:2606. 02010v1 Announce Type: cross Abstract: PlanarBench tests whether LLMs can draw planar graphs as ASCII art given only an edge list -- a spatial reasoning task that resists memorization because edge order, edge orientation, and node labels are all permutable.

By Oleksandr Nikitin

llms benchmarks

arXiv AI

Jun 2

Low-Resource Safety Failures Are Action Failures, Not Representation Failures

arXiv:2606. 01196v1 Announce Type: cross Abstract: Safety alignment learned in high-resource languages transfers poorly to low-resource languages.

By Rashad Aziz, Ikhlasul Akmal Hanif, Fajri Koto

llms benchmarks safety

arXiv AI

Jun 2

Stop Wandering, Find the Keys: LLMs Discriminate Key States for Efficient Multi-Agent Exploration

arXiv:2410. 02511v2 Announce Type: replace Abstract: With expansive state-action spaces, efficient multi-agent exploration remains a longstanding challenge in reinforcement learning.

By Yun Qu, Boyuan Wang, Yuhang Jiang, Jianzhun Shao, Yixiu Mao, Heming Zou, Chang Liu, Cheems Wang, Meiqin Liu, Xiangyang Ji

llms agents reinforcement-learning benchmarks

arXiv AI

Jun 2

STaR-KV: Spatio-Temporal Adaptive Re-weighting for KV Cache Compression in GUI Vision-Language Models

arXiv:2606. 01790v1 Announce Type: cross Abstract: Vision-language-model-based graphical user interface (GUI) agents have shown broad automation capabilities, yet deployment is bottlenecked by a key-value (KV) cache that grows linearly with interaction steps.

By Yuhang Han, Wenzheng Yang, Yujie Chen, Xiangqi Jin, Yaojie Zhang, Siteng Huang, Linfeng Zhang

llms agents efficiency multimodal benchmarks

arXiv AI

Jun 2

Query Circuits: Explaining How Language Models Answer User Prompts

arXiv:2509. 24808v2 Announce Type: replace Abstract: Explaining why a language model produces a particular output requires local, input-level explanations.

By Tung-Yu Wu, Fazl Barez

llms benchmarks

arXiv AI

Jun 2

ReasonBENCH: Benchmarking the (In)Stability of LLM Reasoning

arXiv:2512. 07795v2 Announce Type: replace Abstract: Benchmark scores for LLM reasoning systems are reported as single numbers, yet the same model, strategy, and task can produce meaningfully different answers and costs across repeated executions, even under greedy decoding (T = 0).

By Nearchos Potamitis, Vansh Ramani, Har Ashish Arora, Dhairya Kuchhal, Lars Klein, Akhil Arora

llms benchmarks

arXiv AI

Jun 2

Herculean: An Agentic Benchmark for Financial Intelligence

arXiv:2605. 14355v3 Announce Type: replace Abstract: As AI agents improve, the central question is no longer whether they can solve isolated well-defined financial tasks, but whether they can reliably carry out financial professional work.

By Xueqing Peng, Zhuohan Xie, Yupeng Cao, Haohang Li, Lingfei Qian, Yan Wang, Vincent Jim Zhang, Huan He, Xuguang Ai, Linhai Ma, Ruoyu Xiang, Yueru He, Yi Han, Shuyao Wang, Yuqing Guo, Mingyang Jiang, Yilun Zhao, Youzhong Dong, Xiaoyu Wang, Yankai Chen, Ye Yuan, Qiyuan Zhang, Fuyuan Lyu, Haolun Wu, Yonghan Yang, Zichen Zhao, Yuyang Dai, Fan Zhang, Rania Elbadry, Ayesha Gull, Muhammad Usman Safder, Nuo Chen, Fengbin Zhu, Tianshi Cai, Zimu Wang, Polydoros Giannouris, Yuechen Jiang, Zhiwei Liu, Mohsinul Kabir, Yuyan Wang, Yixiang Zheng, Yangyang Yu, Weijin Liu, Wenbo Cao, Anke Xu, Peng Lu, Jerry Huang, Mingquan Lin, Prayag Tiwari, Yijia Zhao, V\'ictor Guti\'errez-Basulto, Xiao-Yang Liu, Kaleb E Smith, Jiahuan Pei, Arman Cohan, Jimin Huang, Yuehua Tang, Alejandro Lopez-Lira, Xi Chen, Xue Liu, Junichi Tsujii, Jian-Yun Nie, Sophia Ananiadou

agents nlp benchmarks

arXiv AI

Jun 2

MobiBench: Multi-Branch, Modular Benchmark for Mobile GUI Agents

arXiv:2512. 12634v4 Announce Type: replace Abstract: Mobile GUI Agents, AI agents capable of interacting with mobile applications on behalf of users, have the potential to transform human computer interaction.

By Youngmin Im, Byeongung Jo, Jaeyoung Wi, Seungwoo Baek, Tae Hoon Min, Joo Hyung Lee, Sangeun Oh, Insik Shin, Sunjae Lee

agents benchmarks

arXiv AI

Jun 2

VESTA: Visual Exploration with Statistical Tool Agents

arXiv:2606. 00384v1 Announce Type: new Abstract: Fitting quantitative models to data is a central step in scientific workflows, yet it remains one of the least automated.

By William Rudman, Abhishek Divekar, Kanishk Jain, Sebastian Joseph, Stella S. R. Offner, Matthew Lease, Kyle Mahowald, Greg Durrett, Junyi Jessy Li

llms agents multimodal benchmarks

arXiv Machine Learning

Jun 2

Decision-Path Patterns as Tree Reliability Signals: Path-based Adaptive Weighting for Random Forest Classification

arXiv:2605. 20716v5 Announce Type: replace Abstract: Random forests construct each tree with a different, randomised representation of the feature space.

By Youngjoon Park

benchmarks safety

arXiv AI

Jun 2

Evidence-Gated LLM Priors for Multi-Objective Bayesian Optimization

arXiv:2606. 01730v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly used as heuristic advisors for black-box optimization, yet their suggestions and self-reported confidence are not necessarily calibrated to downstream objective values.

By Jiangyu Chen, Banyi

llms benchmarks

arXiv Machine Learning

Jun 2

Inner Product Aware Quantization: Provably Fast, Accurate, and Adaptive Algorithms

arXiv:2606. 00289v1 Announce Type: new Abstract: Quantization is a fundamental tool used to compress datasets, neural network weights, and memory usage in a range of computational tasks.

By Nathan White, Krish Singal

efficiency benchmarks

arXiv AI

Jun 2

Off-the-Shelf LLMs as Process Scorers: Training-Free Alternative to PRMs for Mathematical Reasoning

arXiv:2606. 01682v1 Announce Type: cross Abstract: Selecting the best response from multiple small-model samples using a stronger scorer is a simple inference-time strategy, but fails when the small model has already committed to incorrect reasoning paths.

By Atoosa Chegini, Soheil Feizi

llms reinforcement-learning benchmarks safety

arXiv AI

Jun 2

Boosting Multimodal Federated Learning via Chained Modality Optimization

arXiv:2606. 01856v1 Announce Type: cross Abstract: Multimodal Federated Learning (MMFL) enables privacy-preserving collaborative learning across decentralized clients with heterogeneous data and modality availability.

By Zixin Zhang, Fan Qi, Shuai Li, Xiaoshan Yang, Changsheng Xu

multimodal benchmarks safety

arXiv AI

Jun 2

FlowPlace: Flow Matching for Chip Placement

arXiv:2604. 23658v2 Announce Type: replace-cross Abstract: Chip placement plays an important role in physical design.

By Peng Xie, Ke Xue, Yunqi Shi, Ruo-Tong Chen, Chengrui Gao, Siyuan Xu, Chenjian Ding, Mingxuan Yuan, Chao Qian

diffusion benchmarks

arXiv AI

Jun 2

REALISTA: Realistic Latent Adversarial Attacks that Elicit LLM Hallucinations

arXiv:2605. 12813v2 Announce Type: replace-cross Abstract: Large language models (LLMs) achieve strong performance across many tasks but remain vulnerable to hallucinations, making it important to systematically evaluate their reliability under realistic adversarial inputs.

By Buyun Liang, Jinqi Luo, Liangzu Peng, Kwan Ho Ryan Chan, Darshan Thaker, Kaleab A. Kinfu, Fengrui Tian, Hamed Hassani, Ren\'e Vidal

llms diffusion benchmarks safety

arXiv AI

Jun 2

Beyond AI as Assistants: Toward Autonomous Discovery in Cosmology

arXiv:2605. 14791v2 Announce Type: replace-cross Abstract: Recent advances in artificial intelligence (AI) agents are pushing AI beyond tools toward autonomous scientific discovery.

By Licong Xu, Thomas Borrett

llms agents benchmarks

arXiv AI

Jun 2

OGLS-SD: On-Policy Self-Distillation with Outcome-Guided Logit Steering for LLM Reasoning

arXiv:2605. 12400v2 Announce Type: replace-cross Abstract: We study on-policy self-distillation (OPSD), where a language model improves its reasoning ability by distilling privileged teacher distributions along its own on-policy trajectories.

By Yuxiao Yang, Xiaoyun Wang, Weitong Zhang

llms efficiency benchmarks

arXiv AI

Jun 2

CalArena: A Large-Scale Post-Hoc Calibration Benchmark

arXiv:2605. 30188v2 Announce Type: replace-cross Abstract: Reliable probability estimates are critical in many machine learning applications, yet modern classifiers are often poorly calibrated.

By Eug\`ene Berta, David Holzm\"uller, Francis Bach, Michael I. Jordan

computer-vision benchmarks

arXiv AI

Jun 2

When Single Answer Is Not Enough: Rethinking Single-Step Retrosynthesis Benchmarks for LLMs

arXiv:2602. 03554v2 Announce Type: replace-cross Abstract: Recent progress has expanded the use of large language models (LLMs) in drug discovery, including synthesis planning.

By Bogdan Zagribelnyy, Ivan Ilin, Maksim Kuznetsov, Nikita Bondarev, Mathieu Reymond, Roman Schutski, Thomas MacDougall, Rim Shayakhmetov, Zulfat Miftakhutdinov, Mikolaj Mizera, Vladimir Aladinskiy, Alex Aliper, Alex Zhavoronkov

llms benchmarks

Get the daily briefing