Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

19,246 stories · RSS feed

arXiv AI

Jun 30

HyBIRD: Hyperbolic Bridge Retrieval and Diagnosis for Methodology Inspiration Retrieval

arXiv:2606. 28336v1 Announce Type: cross Abstract: Methodology Inspiration Retrieval (MIR) asks a system to retrieve prior papers whose methods can inspire a new research proposal.

By Yang Yang, Boyun Xu, Hao Fu, Jindong Li, Zining Zhong, Bowen Tian, Jiemin Wu, Menglin Yang, Yutao Yue

llms benchmarks

arXiv AI

Jun 30

When Medical Safety Alignment Fails: A Benchmark for Evaluating LLMs on High-Risk Medical Queries

arXiv:2606. 28332v1 Announce Type: cross Abstract: Large language models (LLMs) are increasingly used for medical and health-related questions, yet their safety in high-risk medical scenarios remains poorly understood.

By Yige Li, Jun Sun, Wei Zhao, Zhe Li, Yutao Wu, Hanxun Huang, Xiang Zheng, Xingjun Ma

llms fine-tuning benchmarks safety

arXiv AI

Jun 30

Improvement of Robot's Simultaneous Localization and Mapping Using an Effective Transformation to Achieve Linear Model

arXiv:2606. 28475v1 Announce Type: cross Abstract: Nowadays mobile robots have wide engineering applications.

By Seyed Farzad Bahreinian, Maziar Palhang, Mohammad Reza Taban, Hasan Enami Eraghi

robotics benchmarks

arXiv AI

Jun 30

HippoSpark: An On-Demand Experience System for LLM Reasoning

arXiv:2606. 29929v1 Announce Type: new Abstract: Distilling historical trajectories into reusable experience to enhance future problem-solving has become a focal point of recent LLM research.

By Jingyao Liu, Danling Meng, Chen Huang, Yukun Yan, Zhenghao Liu, Wenqiang Lei, See-Kiong Ng, Maosong Sun

llms benchmarks

arXiv AI

Jun 30

Managing the Human Fallback: Skill Investment Under Improving AI and Worker Mobility

arXiv:2606. 29111v1 Announce Type: new Abstract: When firms deploy autonomous AI, they must decide how much work to leave to the system and how much to keep workers engaged.

By Simrita Singh, Naireet Ghosh, Tinglong Dai

agents benchmarks

arXiv AI

Jun 30

ScAle: Attention Head Scaling as a Minimal Adapter for Spatial Reasoning in Vision Language Models

arXiv:2606. 29579v1 Announce Type: cross Abstract: Spatial reasoning remains a persistent challenge for many vision language models (VLMs), and improving it typically requires fine-tuning with substantial additional parameters.

By Rahul Chowdhury, Timothy A Rupprecht, Xuan Shen, Pu Zhao, Yanzhi Wang

llms fine-tuning benchmarks

arXiv AI

Jun 30

FD$^2$: A Dedicated Framework for Fine-Grained Dataset Distillation

arXiv:2603. 25144v2 Announce Type: replace-cross Abstract: Dataset distillation (DD) compresses a large training set into a small synthetic set, reducing storage and training cost, and has shown strong results on general benchmarks.

By Hongxu Ma, Guang Li, Shijie Wang, Dongzhan Zhou, Baoli Sun, Takahiro Ogawa, Miki Haseyama, Zhihui Wang

efficiency benchmarks

arXiv Machine Learning

Jun 30

Never Skip a Batch: Dense Learning of Temporal GNNs via Adaptive Pseudo-Supervision

arXiv:2505. 12526v2 Announce Type: replace Abstract: Temporal graph networks suffer from irregular supervision in realworld dynamic graphs, as most minibatches contain few labeled events.

By Alexander Panyshev, Dmitry Vinichenko, Oleg Travkin, Roman Alferov, Alexey Zaytsev

efficiency benchmarks

arXiv Machine Learning

Jun 30

Position: RL Researchers Need to Distinguish Between Solving Simulators and Using Simulators as a Proxy

arXiv:2606. 28433v1 Announce Type: new Abstract: One goal in reinforcement learning (RL) research is to understand general-purpose sequential decision-making, using benchmark simulators as a proxy for learning in deployment settings.

By Matthew Vandergrift, Esraa Elelimy, Martha White

agents reinforcement-learning benchmarks

arXiv AI

Jun 30

CaveAgent: Transforming LLMs into Stateful Runtime Operators

arXiv:2601. 01569v4 Announce Type: replace Abstract: LLM-based agents are increasingly capable of complex task execution, yet current agentic systems remain constrained by text-centric paradigms that struggle with long-horizon tasks due to fragile multi-turn dependencies and context drift.

By Maohao Ran, Zhenglin Wan, Cooper Lin, Yanting Zhang, Hongyu Xin, Hongwei Fan, Yibo Xu, Beier Luo, Yaxin Zhou, Wangbo Zhao, Lijie Yang, Lang Feng, Fuchao Yang, Jingxuan Wu, Yiqiao Huang, Chendong Ma, Yusen Huang, Dailing Jiang, Jianbo Deng, Sirui Han, Yang You, Bo An, Yike Guo, Jun Song

llms agents reinforcement-learning benchmarks

arXiv Machine Learning

Jun 30

TextClusterLab: An Integrated Framework for Reliable Text Clustering Studies

arXiv:2606. 28328v1 Announce Type: cross Abstract: In recent years, text clustering has become a critical technique for applications including intent discovery, topic mining, and recommendation systems.

By Daoming Wan, Yizheng Huang, Jimmy X. Huang

llms rag benchmarks

arXiv AI

Jun 30

Dynamo: Dynamic Skill-Tool Evolution for Vision-Language Agents

arXiv:2606. 30185v1 Announce Type: new Abstract: Improving vision-language models (VLMs) on visual reasoning typically requires retraining or hand-designed prompts and tools.

By Yutao Sun, Yanting Miao, Hao-Xuan Ma, Mengyu Zhou, Mingshuai Chen, Tiancheng Zhao, Dexin Wang, Lei Lv, Li Xu, Xiaoxi Jiang, Guanjun Jiang

llms agents reinforcement-learning multimodal benchmarks

arXiv AI

Jun 30

StarDojo: Benchmarking Open-Ended Behaviors of Agentic Multimodal LLMs in Production-Living Simulations with Stardew Valley

arXiv:2507. 07445v3 Announce Type: replace Abstract: Autonomous agents navigating human society must master both production activities and social interactions, yet existing benchmarks rarely evaluate these skills simultaneously.

By Weihao Tan, Changjiu Jiang, Yu Duan, Mingcong Lei, Jiageng Li, Yitian Hong, Xinrun Wang, Bo An

llms agents robotics multimodal benchmarks

arXiv AI

Jun 30

GraphChase: A Platform and Benchmark for Urban Network Security Games

arXiv:2501. 17559v2 Announce Type: replace Abstract: After the achievement of solving two-player zero-sum games, more AI researchers focus on solving multiplayer games.

By Shuxin Zhuang, Shuxin Li, Tianji Yang, Muheng Li, Xianjie Shi, Bo An, Youzhi Zhang

robotics benchmarks

arXiv AI

Jun 30

Redefining Maritime Anomaly Detection via Equation-Grounded Synthetic Anomalies

arXiv:2606. 29721v1 Announce Type: cross Abstract: Maritime anomaly detection is essential for ensuring maritime safety, security, and efficient traffic management at sea, with Automatic Identification System (AIS) data serving as a primary data source.

By Youngseok Hwang, Sungho Bae, Dohun Lee, Jaeeun Seo, Jeehong Kim, Wonhee Lee, Hyunwoo Park

llms benchmarks safety

arXiv AI

Jun 30

Does Role Specialization Matter for Explanation Faithfulness in Mixture-of-Experts?

arXiv:2606. 29613v1 Announce Type: cross Abstract: Mixture-of-Experts (MoE) architectures have recently been extended with role-based mechanisms for interpretability.

By Yeji Kim, Housam Babiker, Mi-Young Kim, Randy Goebel

diffusion multimodal benchmarks safety

arXiv AI

Jun 30

GROW$^2$: Grounding Which and Where for Robot Tool Use

arXiv:2606. 30632v1 Announce Type: cross Abstract: Can the robot use a plate to cut a cake if no knife is available?

By Yuhong Deng, Yuyao Liu, David Hsu

llms agents robotics multimodal benchmarks

arXiv AI

Jun 30

SABER-Math: Automated Benchmark for Information Retrieval Evaluation in Mathematics

arXiv:2606. 29894v1 Announce Type: cross Abstract: As agentic AI systems tackle more complex mathematical tasks, they increasingly rely on information retrieval (IR) to search problem databases, theorem libraries, and educational resources.

By Nikolay Georgiev, Maria Drencheva, Kseniia Ibragimova, Ivo Petrov, Dimitar I. Dimitrov, Martin Vechev

llms rag agents benchmarks

arXiv AI

Jun 30

Few-Shot Domain Incremental Learning via Continual Vision-Language Consolidation

arXiv:2606. 30190v1 Announce Type: cross Abstract: Existing domain-incremental learning (DIL) strategies call for massive amounts of data to adapt to new domains and suffer from the overfitting problem in the case of data scarcity.

By Naeem Paeedeh, Mahardhika Pratama, Wolfgang Mayer, Mukesh Prasad, Weiping Ding, Yew-Soon Ong

llms diffusion fine-tuning multimodal benchmarks

arXiv AI

Jun 30

Self-Organized Conformal Prediction: Reducing Regional Coverage Gaps with Unsupervised Group Discovery

arXiv:2606. 29403v1 Announce Type: cross Abstract: Conformal prediction guarantees marginal coverage, but pooled calibration averages over heterogeneous regions and can mask regional undercoverage in safety-critical subgroups.

By Louis Berthier, Ahmed Shokry, Maxime Moreaud, Guillaume Ramelet, Aymeric Dieuleveut

rag benchmarks safety

Get the daily briefing