Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

16,065 stories · RSS feed

arXiv AI

Jul 22

Doctorina MedBench-ICD10: A Dialogue-Based Benchmark and Evaluation Framework for Agent-Based Medical AI

arXiv:2603. 25821v2 Announce Type: replace-cross Abstract: We present Doctorina MedBench, a comprehensive evaluation framework for agent-based medical AI based on the simulation of realistic physician-patient interactions.

By Anna Kozlova, Stanislau Salavei, Pavel Satalkin, Hanna Plotnitskaya, Sergey Parfenyuk

agents benchmarks safety

arXiv AI

Jul 22

Towards Principled Continual Anomaly Detection: A Systematic Framework and Benchmark Scenarios

arXiv:2607. 18289v1 Announce Type: cross Abstract: Continual anomaly detection (CAD) studies how models can adapt to evolving data distributions while retaining performance on previously observed regimes.

By Kamil Faber, Mateusz Smendowski, Roberto Corizzo

benchmarks

arXiv AI

Jul 22

Dr. Zero: Self-Evolving Search Agents without Training Data

arXiv:2601. 07055v2 Announce Type: replace Abstract: As high-quality data becomes increasingly difficult to obtain, self-evolution without curated training data has emerged as a promising paradigm.

By Zhenrui Yue, Kartikeya Upasani, Xianjun Yang, Suyu Ge, Shaoliang Nie, Yuning Mao, Zhe Liu, Dong Wang

llms agents nlp benchmarks

arXiv Machine Learning

Jul 22

EmoEUS: Uncertainty Supervision for Multimodal Emotion Recognition in Conversation

arXiv:2607. 18336v1 Announce Type: cross Abstract: Multimodal emotion recognition in conversation (MERC) can leverage multimodal and contextual cues to boost recognition performance.

By Zilong Huang, Kong Aik Lee, Junjie Li, Zhe Li, Man-Wai Mak

multimodal benchmarks

arXiv AI

Jul 22

CPGRec+: A Balance-oriented Framework for Personalized Video Game Recommendations

arXiv:2604. 14586v3 Announce Type: replace-cross Abstract: The rapid expansion of gaming industry requires advanced recommender systems tailored to its dynamic landscape.

By Xiping Li, Aier Yang, Jianghong Ma, Kangzhe Liu, Shanshan Feng, Haijun Zhang, Yi Zhao

llms benchmarks

arXiv Machine Learning

Jul 22

One Model, Many Graphs: Learning over Attributed Graphs across Heterogeneous Modalities with Vision-Language Models

arXiv:2607. 19128v1 Announce Type: new Abstract: Vision-language models (VLMs) provide a unified representation space for textual and visual information, yet their potential as general-purpose backbones for graph-structured data remains largely unexplored.

By Jiayi Yang, Yifang Chen, Yuanfu Sun, Jiajin Liu, Qiaoyu Tan

llms rag multimodal benchmarks

arXiv AI

Jul 22

SKETCH: Semantic Key-Point Conditioning for Long-Horizon Vessel Trajectory Prediction

arXiv:2601. 18537v4 Announce Type: replace-cross Abstract: Accurate long-horizon vessel trajectory prediction remains challenging due to compounded uncertainty from complex navigation behaviors and environmental factors.

By Linyong Gan, Zimo Li, Wenxin Xu, Xingjian Li, Jianhua Z. Huang, Enmei Tu, Shuhang Chen

benchmarks

arXiv AI

Jul 22

Spectral Higher-Order Neural Networks Have Sharp Expressivity Bounds

arXiv:2607. 19042v1 Announce Type: cross Abstract: Neural hypergraphs are a natural generalization of neural networks, the reference models in modern machine learning.

By Gianluca Peri, Diego Febbe, Duccio Fanelli

benchmarks safety

arXiv AI

Jul 22

FVRuleLearner: Operator-Level Reasoning Tree (Op-Tree)-Based Rules Learning for Formal Verification

arXiv:2604. 03245v2 Announce Type: replace-cross Abstract: The remarkable reasoning and code generation capabilities of large language models (LLMs) have recently motivated increasing interest in automating formal verification (FV), a process that ensures hardware correctness through mathematically precise assertions but remains highly labor-intensive, particularly through the translation of natural language into SystemVerilog Assertions (NL-to-SVA).

By Lily Jiaxin Wan, Chia-Tung Ho, Yunsheng Bai, Cunxi Yu, Ghaith Bany Hamad, Deming Chen, Haoxing Ren

llms benchmarks safety

arXiv AI

Jul 22

CoGoal3D: Collaborative 3D Object Detection with 3D-Aware Fusion and Refinement

arXiv:2607. 19036v1 Announce Type: cross Abstract: V2X collaborative object detection features overcoming the limitations of single-vehicle systems by aggregating environmental features from multiple collaborative agents.

By Zhihao Yang, Zhiyu Xiang, Peng Xu, Tianyu Pu, Kai Wang, Eryun Liu, Dongping Zhang, Yong Ding

agents computer-vision benchmarks

arXiv AI

Jul 22

GUIDED Network-Agnostic Feature Initialization for Spatial Transferability in GNN-based Models

arXiv:2607. 19270v1 Announce Type: cross Abstract: The Traffic Assignment Problem is a fundamental but computationally expensive component of transportation planning.

By Alessandro Scalese, Santhanakrishnan Narayanan, Constantinos Antoniou

rag fine-tuning multimodal benchmarks

arXiv AI

Jul 22

Quantum Cryptanalysis on IBM Quantum Hardware: Extending Even--Mansour Period Recovery from $N=4$ to $N=10$

arXiv:2607. 18340v1 Announce Type: cross Abstract: We report genuine-un-compiled, textbook-faithful-quantum cryptanalysis of symmetric-cipher structures executed on real IBM quantum hardware (ibm\_kingston, Heron generation).

By Taebong Kim, Youngsik Hong, Minsik Kim, Sunyoung Choi, Jaewon Jang, Junghoon Shin, Minseo Kim

benchmarks

arXiv Machine Learning

Jul 22

Vectorizing the Trie: Efficient Constrained Decoding for LLM-based Generative Retrieval on Accelerators

arXiv:2602. 22647v2 Announce Type: replace-cross Abstract: Generative retrieval has emerged as a powerful paradigm for LLM-based recommendation.

By Zhengyang Su, Isay Katsman, Yueqi Wang, Ruining He, Lukasz Heldt, Raghunandan Keshavan, Shao-Chuan Wang, Xinyang Yi, Mingyan Gao, Onkar Dalal, Lichan Hong, Ed Chi, Ningren Han

llms efficiency benchmarks

arXiv AI

Jul 22

FormGym: Doing Paperwork with Agents

arXiv:2506. 14079v4 Announce Type: replace Abstract: Completing paperwork is a challenging and time-consuming problem.

By Matthew Toles, Rattandeep Singh, Isaac Song, Zhou Yu

llms agents computer-vision multimodal benchmarks

arXiv AI

Jul 22

CodeRescue: Budget-Calibrated Recovery Routing for Coding Agents

arXiv:2607. 19338v1 Announce Type: new Abstract: Coding agents increasingly operate in executable environments where a failed attempt produces actionable feedback rather than merely an incorrect answer.

By Qijia He, Jiayi Cheng, Chenqian Le, Rui Wang, Xunmei Liu, Yixian Chen, Jie Mei, Zhihao Wang, Xupeng Chen, Yuhuan Chen, Tao Wang

llms agents benchmarks

arXiv Machine Learning

Jul 22

Weak-to-Strong Learning in Decision Making

arXiv:2607. 18467v1 Announce Type: new Abstract: Many operational decisions rely on predictive models that estimate uncertain outcomes conditional on observable contexts.

By Jingwei Ji, Renyuan Xu

benchmarks

arXiv AI

Jul 22

MEDIC: Comprehensive Evaluation of Leading Indicators for LLM Safety and Utility in Clinical Applications

arXiv:2409. 07314v3 Announce Type: replace-cross Abstract: While Large Language Models (LLMs) achieve superhuman performance on standardized medical licensing exams, these static benchmarks have become saturated and increasingly disconnected from the functional requirements of clinical workflows.

By Praveenkumar Kanithi, Cl\'ement Christophe, Marco AF Pimentel, Tathagata Raha, Prateek Munjal, Nada Saadi, Hamza A Javed, Svetlana Maslenkova, Nasir Hayat, Ronnie Rajan, Shadab Khan

llms fine-tuning benchmarks safety data-engineering

arXiv AI

Jul 22

Off-Context GRPO: Learning to Reason on Hard Problems using Privileged Information

arXiv:2607. 19313v1 Announce Type: cross Abstract: Reinforcement learning with verifiable rewards (RLVR) improves reasoning in large language models.

By Priyank Agrawal, Ankur Samanta, Shervin Ghasemlou, Jalaj Bhandari, Kavosh Asadi, Daniel Jiang, Aditya Modi

llms reinforcement-learning benchmarks

arXiv AI

Jul 22

Now We Know? A Systematic Comparison of TerraMind and THOR

arXiv:2607. 18504v1 Announce Type: cross Abstract: Benchmarks for Geospatial Foundation Models (GFMs) increasingly rank models by aggregate score, but such rankings obscure why models differ: how much of the gap is architecture, how much is decoder capacity, and how much is a use-case-specific artefact?

By Frederick Schindlegger, Kenzo Bounegta, Eva Gmelich Meijling, Johannes Jakubik, Arnt-B{\o}rre Salberg, Theodor Forgaard, Nicolas Longepe, Valerio Marsocci

computer-vision fine-tuning multimodal benchmarks

arXiv AI

Jul 22

ProbSPARQL: Querying Knowledge Graphs with Multi-dimensional, Uncertain Numeric Data

arXiv:2607. 18262v1 Announce Type: new Abstract: The SFB 1574 Circular Factory is building a shared knowledge graph infrastructure for integrating data about returned products.

By Jingcheng Wu, Ratan Bahadur Thapa, Daniel Hernandez, Hongkuan Zhou, Steffen Staab

benchmarks

Get the daily briefing