Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

19,246 stories · RSS feed

arXiv AI

Jun 30

SABER-Math: Automated Benchmark for Information Retrieval Evaluation in Mathematics

arXiv:2606. 29894v1 Announce Type: cross Abstract: As agentic AI systems tackle more complex mathematical tasks, they increasingly rely on information retrieval (IR) to search problem databases, theorem libraries, and educational resources.

By Nikolay Georgiev, Maria Drencheva, Kseniia Ibragimova, Ivo Petrov, Dimitar I. Dimitrov, Martin Vechev

llms rag agents benchmarks

arXiv AI

Jun 30

Self-Organized Conformal Prediction: Reducing Regional Coverage Gaps with Unsupervised Group Discovery

arXiv:2606. 29403v1 Announce Type: cross Abstract: Conformal prediction guarantees marginal coverage, but pooled calibration averages over heterogeneous regions and can mask regional undercoverage in safety-critical subgroups.

By Louis Berthier, Ahmed Shokry, Maxime Moreaud, Guillaume Ramelet, Aymeric Dieuleveut

rag benchmarks safety

arXiv AI

Jun 30

Does Role Specialization Matter for Explanation Faithfulness in Mixture-of-Experts?

arXiv:2606. 29613v1 Announce Type: cross Abstract: Mixture-of-Experts (MoE) architectures have recently been extended with role-based mechanisms for interpretability.

By Yeji Kim, Housam Babiker, Mi-Young Kim, Randy Goebel

diffusion multimodal benchmarks safety

arXiv AI

Jun 30

GROW$^2$: Grounding Which and Where for Robot Tool Use

arXiv:2606. 30632v1 Announce Type: cross Abstract: Can the robot use a plate to cut a cake if no knife is available?

By Yuhong Deng, Yuyao Liu, David Hsu

llms agents robotics multimodal benchmarks

arXiv Machine Learning

Jun 30

CAREBench: A Child-Safety Risk Benchmark for Language Models

arXiv:2606. 29685v1 Announce Type: new Abstract: How can we evaluate whether frontier AI systems recognize child-safety risks before they escalate into explicit harm?

By Kaavya Krishna-Kumar, Elaine Lau, Vaughn Robinson, Jay Caldwell, Sheriff Issaka, Skyler Wang, Francisco Guzm\'an, Steven Kelling, Jonas Mueller

llms benchmarks safety

arXiv Machine Learning

Jun 30

Blackknife: Hard-Label Query-Limited Black-Box Attacks on Heterogeneous Graph Neural Networks

arXiv:2606. 29240v1 Announce Type: new Abstract: Heterogeneous graph neural networks (HGNNs) have achieved strong performance in modeling complex graph-structured data with multiple node and relation types.

By Honglin Gao, Junhao Ren, Lan Zhao, Yue Yang, Jindong Chang, Gaoxi Xiao

benchmarks safety

arXiv Machine Learning

Jun 30

TextClusterLab: An Integrated Framework for Reliable Text Clustering Studies

arXiv:2606. 28328v1 Announce Type: cross Abstract: In recent years, text clustering has become a critical technique for applications including intent discovery, topic mining, and recommendation systems.

By Daoming Wan, Yizheng Huang, Jimmy X. Huang

llms rag benchmarks

arXiv Machine Learning

Jun 30

Never Skip a Batch: Dense Learning of Temporal GNNs via Adaptive Pseudo-Supervision

arXiv:2505. 12526v2 Announce Type: replace Abstract: Temporal graph networks suffer from irregular supervision in realworld dynamic graphs, as most minibatches contain few labeled events.

By Alexander Panyshev, Dmitry Vinichenko, Oleg Travkin, Roman Alferov, Alexey Zaytsev

efficiency benchmarks

arXiv AI

Jun 30

HippoSpark: An On-Demand Experience System for LLM Reasoning

arXiv:2606. 29929v1 Announce Type: new Abstract: Distilling historical trajectories into reusable experience to enhance future problem-solving has become a focal point of recent LLM research.

By Jingyao Liu, Danling Meng, Chen Huang, Yukun Yan, Zhenghao Liu, Wenqiang Lei, See-Kiong Ng, Maosong Sun

llms benchmarks

arXiv AI

Jun 30

ADEPT: An Entropy-Driven Dual-Strategy Agent for Interactive Video Retrieval

arXiv:2606. 28326v1 Announce Type: cross Abstract: This research aims to solve the challenge of video retrieval from massive datasets, caused by ambiguous user queries.

By Ke Chen, Shengyuan Han, Yongfeng Huang, Yujin Zhu, Jingwei Xiong, Liang Xu, Jundong Liu

llms agents benchmarks

arXiv AI

Jun 30

Redefining Maritime Anomaly Detection via Equation-Grounded Synthetic Anomalies

arXiv:2606. 29721v1 Announce Type: cross Abstract: Maritime anomaly detection is essential for ensuring maritime safety, security, and efficient traffic management at sea, with Automatic Identification System (AIS) data serving as a primary data source.

By Youngseok Hwang, Sungho Bae, Dohun Lee, Jaeeun Seo, Jeehong Kim, Wonhee Lee, Hyunwoo Park

llms benchmarks safety

arXiv AI

Jun 30

GraphChase: A Platform and Benchmark for Urban Network Security Games

arXiv:2501. 17559v2 Announce Type: replace Abstract: After the achievement of solving two-player zero-sum games, more AI researchers focus on solving multiplayer games.

By Shuxin Zhuang, Shuxin Li, Tianji Yang, Muheng Li, Xianjie Shi, Bo An, Youzhi Zhang

robotics benchmarks

arXiv AI

Jun 30

StarDojo: Benchmarking Open-Ended Behaviors of Agentic Multimodal LLMs in Production-Living Simulations with Stardew Valley

arXiv:2507. 07445v3 Announce Type: replace Abstract: Autonomous agents navigating human society must master both production activities and social interactions, yet existing benchmarks rarely evaluate these skills simultaneously.

By Weihao Tan, Changjiu Jiang, Yu Duan, Mingcong Lei, Jiageng Li, Yitian Hong, Xinrun Wang, Bo An

llms agents robotics multimodal benchmarks

arXiv AI

Jun 30

Assessing the Business Process Modeling Competences of Large Language Models

arXiv:2601. 21787v2 Announce Type: replace-cross Abstract: The creation of Business Process Model and Notation (BPMN) models is a complex and time-consuming task requiring both domain knowledge and proficiency in modeling conventions.

By Chantale Lauer, Peter Pfeiffer, Alexander Rombach, Nijat Mehdiyev

llms fine-tuning benchmarks

arXiv AI

Jun 30

FD$^2$: A Dedicated Framework for Fine-Grained Dataset Distillation

arXiv:2603. 25144v2 Announce Type: replace-cross Abstract: Dataset distillation (DD) compresses a large training set into a small synthetic set, reducing storage and training cost, and has shown strong results on general benchmarks.

By Hongxu Ma, Guang Li, Shijie Wang, Dongzhan Zhou, Baoli Sun, Takahiro Ogawa, Miki Haseyama, Zhihui Wang

efficiency benchmarks

arXiv Machine Learning

Jun 30

Position: RL Researchers Need to Distinguish Between Solving Simulators and Using Simulators as a Proxy

arXiv:2606. 28433v1 Announce Type: new Abstract: One goal in reinforcement learning (RL) research is to understand general-purpose sequential decision-making, using benchmark simulators as a proxy for learning in deployment settings.

By Matthew Vandergrift, Esraa Elelimy, Martha White

agents reinforcement-learning benchmarks

arXiv AI

Jun 30

CaveAgent: Transforming LLMs into Stateful Runtime Operators

arXiv:2601. 01569v4 Announce Type: replace Abstract: LLM-based agents are increasingly capable of complex task execution, yet current agentic systems remain constrained by text-centric paradigms that struggle with long-horizon tasks due to fragile multi-turn dependencies and context drift.

By Maohao Ran, Zhenglin Wan, Cooper Lin, Yanting Zhang, Hongyu Xin, Hongwei Fan, Yibo Xu, Beier Luo, Yaxin Zhou, Wangbo Zhao, Lijie Yang, Lang Feng, Fuchao Yang, Jingxuan Wu, Yiqiao Huang, Chendong Ma, Yusen Huang, Dailing Jiang, Jianbo Deng, Sirui Han, Yang You, Bo An, Yike Guo, Jun Song

llms agents reinforcement-learning benchmarks

arXiv AI

Jun 30

CostBench: Evaluating Multi-Turn Cost-Optimal Planning and Adaptation in Dynamic Environments for LLM Tool-Use Agents

arXiv:2511. 02734v3 Announce Type: replace Abstract: Current evaluations of Large Language Model (LLM) agents primarily emphasize task completion, often overlooking resource efficiency and adaptability.

By Jiayu Liu, Cheng Qian, Zhaochen Su, Qing Zong, Shijue Huang, Bingxiang He, Yi R. Fung

llms agents benchmarks

arXiv Machine Learning

Jun 30

Friend or Foe

arXiv:2509. 00123v2 Announce Type: replace-cross Abstract: A fundamental challenge in microbial ecology is determining whether bacteria compete or cooperate in different environmental conditions.

By Oleksandr Cherednichenko, Josephine Solowiej-Wedderburn, Laura M. Carroll, Eric Libby

benchmarks

arXiv AI

Jun 30

HMARS: A Hierarchical Multi-Agent Memory System for Long-Context Reasoning

arXiv:2606. 28349v1 Announce Type: cross Abstract: Long-context reasoning requires models to access, retrieve, and integrate evidence scattered across documents, dialogues, and accumulated interaction histories.

By Zeju Li, Ziyang Zheng, Yizhou Zhou, Qiang Xu

rag agents benchmarks

Get the daily briefing