Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

17,067 stories · RSS feed

arXiv Machine Learning

Jul 7

CLARITY: Medical World Model for Guiding Treatment Decisions by Modeling Context-Aware Disease Trajectories in Latent Space

arXiv:2512. 08029v3 Announce Type: replace Abstract: Clinical decision-making in oncology requires predicting dynamic disease evolution, a task current static AI predictors cannot perform.

By Tianxingjian Ding, Yuanhao Zou, Chen Chen, Mubarak Shah, Yu Tian

llms diffusion benchmarks

arXiv Machine Learning

Jul 7

CDCP: Conditional Diffusion Model with Contextual Prompts for Multi-task Offline Safe Reinforcement Learning

arXiv:2607. 03903v1 Announce Type: new Abstract: Multi-task offline safe reinforcement learning (RL) promises to learn a shared optimal safe policy from offline data across multiple tasks.

By Jiayi Guan, Tianle Zhang, Li Shen, Ruiqi Zhang, Ao Zhou, Lusong Li, Guai Chen, Mengjie Li, Alois Knoll, Xiaodong He, Changjun Jiang

llms diffusion reinforcement-learning benchmarks safety

arXiv Machine Learning

Jul 7

Measuring What Matters: A Unified Evaluation Framework for GNN Explainability

arXiv:2607. 04600v1 Announce Type: new Abstract: Graph eXplainable AI (G-XAI) is increasingly important for making Graph Neural Networks interpretable and accountable.

By Francesco Paolo Nerini, Mirko Zaffaroni, Paolo Baracco, Gabriele Ciravegna, Alan Perotti

benchmarks

arXiv AI

Jul 7

Embodied Operators and Benchmarking: Toward Reusable and Deployable Embodied Intelligence Systems

arXiv:2607. 03283v1 Announce Type: new Abstract: Embodied intelligence systems require not only end-to-end policy models, but also reusable functional modules that transform multimodal observations, robot states, human demonstrations, and task contexts into structured representations, decisions, trajectories, control references, and system services.

By Junwu Xiong, Jiaxuan Gao, Wei Chai, Renxing Chen, Yuzhen Li, Yu Guo, Yucheng Guo, Mingxi Luo, Wenyang Ma, Yiyun Mou, Yifei Zhang, Chen Zhou, Yongjian Guo

computer-vision robotics efficiency multimodal benchmarks safety

arXiv AI

Jul 7

When Simpler Is Better: Evaluating Translation Pipelines for Medieval Latin Manuscripts

arXiv:2607. 03836v1 Announce Type: cross Abstract: Despite remarkable progress in machine translation, Vision Language Models (VLMs) struggle on historical manuscripts, a domain that stresses core Natural Language Processing (NLP) capabilities: low-resource transliteration, archaic vocabulary, and noisy input signals.

By Nguyen Kim Hai Bui, Md. Easin Arafat, Tam\'as G\'abor Orosz, Mufti Mahmud

llms rag computer-vision nlp multimodal benchmarks

arXiv AI

Jul 7

AgentGym2: Benchmarking Large Language Model Agents in De-Idealized Real-World Environments

arXiv:2607. 05174v1 Announce Type: new Abstract: Language agents, i.

By Zhiheng Xi, Dingwen Yang, Jiaqi Liu, Jixuan Huang, Honglin Guo, Baodai Huang, Tinggang Chen, Qi Zhang, Zhonghang Lu, Chenyu Liu, Jiajun Sun, Jiazheng Zhang, Dingwei Zhu, Xin Guo, Junzhe Wang, Zhihao Zhang, Yuming Yang, Junjie Ye, Minghe Gao, Dongrui Liu, Jiaming Ji, Guohao Li, Tao Gui, Qi Zhang, Xuanjing Huang

llms agents benchmarks

arXiv AI

Jul 7

Punching Above Their Weight: Classification-Head Fine-Tuning of Tiny Language Models (TLMs) for Verifiable Multiple-Choice Tasks

arXiv:2607. 03801v1 Announce Type: cross Abstract: We define Tiny Language Models (TLMs) as models below roughly 3B parameters that fit on mainstream consumer devices.

By Bhavesh Sood, Jaromir Savelka

llms fine-tuning benchmarks

arXiv AI

Jul 7

EventCoT: Event-centric Video Chain-of-thought for Reasoning Temporal Localization

arXiv:2607. 04872v1 Announce Type: cross Abstract: Reasoning temporal localization (RTL) requires a model to generate an answer that itself contains the time interval supporting it, so high-level reasoning and precise temporal grounding must be produced jointly in a single response.

By Youngkil Song, Yoonjae Baek, Dongwon Kim, Inho Kim, Dongkeun Kim, Suha Kwak

rag nlp benchmarks

arXiv Machine Learning

Jul 7

Few-Step Diffusion Language Models via Trajectory Self-Distillation

arXiv:2602. 12262v4 Announce Type: replace-cross Abstract: Diffusion large language models (DLLMs) have emerged as powerful generative models with the promise of fast text generation through parallel decoding.

By Tunyu Zhang, Xinxi Zhang, Ligong Han, Haizhou Shi, Xiaoxiao He, Zhuowei Li, Hao Wang, Kai Xu, Akash Srivastava, Chengzhi Mao, Hao Wang, Vladimir Pavlovic, Dimitris N. Metaxas

llms diffusion efficiency benchmarks

arXiv Machine Learning

Jul 7

Scalable Dexterous Robot Learning with AR-based Remote Human-Robot Interactions

arXiv:2602. 07341v2 Announce Type: replace Abstract: This paper focuses on the scalable robot learning for manipulation in the dexterous robot arm-hand systems, where the remote human-robot interactions via augmented reality (AR) are established to collect the expert demonstration data for improving efficiency.

By Yicheng Yang, Ruijiao Li, Lifeng Wang, Shuai Zheng, Shunzheng Ma, Keyu Zhang, Tuoyu Sun, Chenyun Dai, Jie Ding, Zhuo Zou

reinforcement-learning robotics benchmarks safety

arXiv AI

Jul 7

PDEFlow: Autonomous Agentic PDE Pipelines for Neural Operator Learning and Solver-Free Inference

arXiv:2607. 05134v1 Announce Type: cross Abstract: We present PDEFlow, an autonomous agentic framework that turns user-level ODE and PDE descriptions into solver-backed neural-operator pipelines.

By Akshat Jani, Prathamesh Gadekar, Sakhinana Sagar Srinivas, Venkataramana Runkana

agents benchmarks

arXiv AI

Jul 7

MemPose: Category-level Object Pose Estimation with Memory

arXiv:2607. 04930v1 Announce Type: cross Abstract: In the pursuit of robust and generalizable category-level object pose estimation, most existing methods adopt parametric formulations that learn effective representations from data, yet they primarily encode category-level patterns into fixed shape priors or static parameter weights, which limits their scalability to highly diverse instances.

By Xiao Lin, Minghao Zhu, Yun Peng, Liuyi Wang, Qiyi Wang, Chengju Liu, Qijun Chen

computer-vision benchmarks

arXiv Machine Learning

Jul 7

GRAFT: Grafted Reference Audio for Fine-grained Pronunciation in Zero-shot Text-to-Speech

arXiv:2607. 02633v1 Announce Type: new Abstract: We present GRAFT, a per-word pronunciation conditioning mechanism for text-to-speech neural codec language modeling.

By Antonis Asonitis, Francesco Verdini, Aref Farhadipour, Vijeta Avijeet, Pierre-Edouard Honnet, Marzieh Razavi, Juan Pablo Zuluaga Gomez

nlp multimodal benchmarks

arXiv AI

Jul 7

Estimating Individual Tree Height and Species from UAV Imagery

arXiv:2603. 23669v2 Announce Type: replace-cross Abstract: Accurate estimation of forest biomass, a major carbon sink, relies heavily on tree-level traits such as height and species.

By Jannik Endres, Etienne Lalibert\'e, David Rolnick, Arthur Ouaknine

benchmarks

arXiv AI

Jul 7

Test-Time Adaptation for EEG Foundation Models: A Systematic Study under Real-World Distribution Shifts

arXiv:2604. 16926v2 Announce Type: replace-cross Abstract: Electroencephalography (EEG) foundation models have shown strong potential for learning generalizable representations from large-scale neural data, yet their clinical deployment is hindered by distribution shifts across clinical settings, devices, and populations.

By Gabriel Jason Lee, Jathurshan Pradeepkumar, Jimeng Sun

benchmarks safety

arXiv AI

Jul 7

PuzzleMoE: Efficient Compression of Large Mixture-of-Experts Models via Sparse Expert Merging and Bit-packed inference

arXiv:2511. 04805v2 Announce Type: replace-cross Abstract: Mixture-of-Experts (MoE) models have shown strong potential in scaling language models efficiently by activating only a small subset of experts per input.

By Yushu Zhao, Zheng Wang, Minjia Zhang

llms benchmarks

arXiv Machine Learning

Jul 7

Physics-Informed Graph Learning with Uncertainty Awareness for Open-Set Domain Generalization in Fault Diagnosis

arXiv:2607. 04188v1 Announce Type: new Abstract: Intelligent industrial maintenance critically relies on reliable fault diagnosis of rotating machinery.

By Jinfeng Zhu, Shiyu Long, Ye Yuan

benchmarks

arXiv AI

Jul 7

When is a System Discoverable from Data? Discovery Requires Chaos

arXiv:2511. 08860v2 Announce Type: replace-cross Abstract: The deep learning revolution has spurred a rise in advances of using AI in sciences.

By Zakhar Shumaylov, Peter Zaika, Philipp Scholl, Gitta Kutyniok, Lior Horesh, Carola-Bibiane Sch\"onlieb

benchmarks

arXiv Machine Learning

Jul 7

F-ACVAE: A Federated Adaptive Conditional Variational Auto-Encoder for Privacy-Preserving Intrusion Detection in IoT Networks

arXiv:2607. 04698v1 Announce Type: new Abstract: The rapid proliferation of Internet of things (IoT) devices has significantly expanded the cyber-attack surface, necessitating robust and privacy-preserving intrusion detection systems (IDS).

By Mohammad Ansarimehr, Somayeh Changiz, Ehsan Baghishani, Ali Mousavi

benchmarks safety

arXiv AI

Jul 7

An Exploration of Agentic Information Fusion for Test Maintenance Prediction

arXiv:2607. 04786v1 Announce Type: cross Abstract: Test maintenance is a critical, yet costly, activity - particularly as codebases rapidly evolve.

By Jingxiong Liu, Nasser Mohammadiha, Gregory Gay

agents benchmarks

Get the daily briefing