Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

16,906 stories · RSS feed

arXiv AI

Jul 16

CoDiffGRN: Rethinking Gene Regulatory Network Inference via the BEELINE-KGC Benchmark and Co-evolutionary Discrete Diffusion

arXiv:2607. 13120v1 Announce Type: cross Abstract: Inferring gene regulatory networks (GRNs) from single-cell transcriptomic data is crucial for biological discovery, yet existing approaches suffer from a fundamental misalignment with real-world needs.

By Jiaze Song, Runhao Zhao, Minghao Xu, Bin Cui, Wentao Zhang

diffusion benchmarks

arXiv Machine Learning

Jul 16

HRIBench: Benchmarking Interaction-Centric Human-Robot Collaboration

arXiv:2607. 13056v1 Announce Type: cross Abstract: Current vision-language-action (VLA) benchmarks primarily evaluate isolated manipulation skills while leaving human-robot interaction structure largely unmodeled.

By Chang Liu, Jiawei Zhang, Tao Zhang, Ye Wang, Hongyu Zhou, Qin Jin

agents robotics fine-tuning multimodal benchmarks safety

arXiv AI

Jul 16

Boogu-Image-0.1: Boosting Open-Source Unified Multimodal Understanding and Generation

arXiv:2607. 13125v1 Announce Type: cross Abstract: We introduce Boogu-Image-0.

By Guoxuan Chen, Chufeng Xiao, Haoran Yang, Siyue Xie, Binxiao Huang, Ming Zhang, Cheuk Him Chau, Xinyu Fu, Yingzhao Lian, Tom S. Y. Li, Jintao Lin, Bowen Dong, Zian Qian, Yuhao Liu, Yuxuan Hu, Weikang Shi, Bin Zou, Bowen Zheng, Haoxuan Che, Chang Chen, Yuyang He, Heyang Sun, Tianyu Huang, Chong Hou Choi, Cheng Gong, Han Shi, Haoli Bai, Xihui Liu, Hongsheng Li, Qifeng Chen, Chao Huang, Rui Liu, Chenyang Lei

llms agents diffusion multimodal benchmarks data-engineering

arXiv AI

Jul 16

ShortOPD: Recovering Pruned LLMs with Short-to-Long On-Policy Distillation

arXiv:2607. 13124v1 Announce Type: cross Abstract: Structured pruning is a hardware-friendly way to compress LLMs, but it is mostly validated on multiple-choice recognition tasks, while the same compressed checkpoints can collapse on the free-form generation that deployment actually requires.

By Qingyu Zhang, Qianhao Yuan, Hongyu Lin, Yaojie Lu, Xianpei Han, Le Sun, Xiang Li, Ming Xu, Jiarui Li, Xiuyin Zhao

llms fine-tuning efficiency benchmarks

arXiv AI

Jul 16

Disentangling Knowledge States with Ability and Proficiency Modeling for Knowledge Tracing

arXiv:2607. 13103v1 Announce Type: cross Abstract: Knowledge tracing (KT) aims to predict students' future performance by modeling their evolving knowledge states from historical interactions.

By Duantengchuan Li, Yingqian Bi, Jinsong Chen, Rui Zhang, Mingwen Tong

llms benchmarks safety

arXiv AI

Jul 16

Accuracy Without Grounding: Diagnosing Visual Dependency Dissociation in Video LLM Benchmarks

arXiv:2607. 13305v1 Announce Type: cross Abstract: Benchmark accuracy in video large language models (LLMs) is often treated as evidence of visual understanding.

By Jae Joong Lee

llms benchmarks

arXiv AI

Jul 16

MxGPS: Multiplex Graph Transformers for a Power Grid Foundation Model

arXiv:2607. 13763v1 Announce Type: cross Abstract: Single-task fine-tuning of graph neural networks (GNNs) for power grid problems exhibits a systematic failure mode: models that achieve the lowest in-distribution error degrade the most under topology shift.

By Charilaos Papaioannou, Ioannis Tsantilas, Dimitris Giannakakos, Vasilis Michalakopoulos, Sotiris Pelekis, Vangelis Marinakis, Arsam Aryandoust, Antonello Monti, Ricardo J. Bessa, Perdo P. Vergara, Jochen Cremer, Elissaios Sarmas

llms fine-tuning benchmarks

arXiv AI

Jul 16

How Far Can Root Cause Analysis Go on Real-World Telemetry Data?

arXiv:2607. 13548v1 Announce Type: new Abstract: Identifying root causes in production microservice failures requires reasoning over large-scale, multimodal telemetry spanning metrics, logs, and traces, a problem that has proved resistant to both classical and LLM-based approaches.

By Athira Gopal, Ashwanth Krishnan

llms agents multimodal benchmarks

arXiv Machine Learning

Jul 16

Foundation Models for Credit Risk Prediction: A Game Changer?

arXiv:2605. 18147v2 Announce Type: replace Abstract: Predictive models play a pivotal role in credit risk management, guiding critical decisions through accurate estimation of default probabilities and losses.

By Bart Baesens, Andreas Goethals, Stefan Lessmann, Simon De Vos, Cristi\'an Bravo, David Martens, Victor Medina-Olivares, Christophe Mues, Maria Oskarsd\'ottir, Seppe vanden Broucke, Tony Van Gestel, Tim Verdonck, Wouter Verbeke

llms computer-vision nlp benchmarks

arXiv Machine Learning

Jul 16

Test-Time Learning with an Evolving Library

arXiv:2605. 14477v2 Announce Type: replace Abstract: We introduce EvoLib, a test-time learning framework that enables large language models to accumulate, reuse, and evolve knowledge across problem instances without parameter updates or external supervision.

By Weijia Xu, Alessandro Sordoni, Chandan Singh, Zelalem Gero, Michel Galley, Xingdi Yuan, Jianfeng Gao

llms agents benchmarks

arXiv AI

Jul 16

AgentCompass: A Unified Evaluation Infrastructure for Agent Capabilities

arXiv:2607. 13705v1 Announce Type: new Abstract: As Large Language Models (LLMs) evolve into autonomous agents, the need for unified evaluation infrastructure becomes critical.

By Zichen Ding, Jiaye Ge, Shufan Jiang, Kai Chen, Mo Li, Qingqiu Li, Zehao Li, Zonglin Li, Tiaohao Liang, Shudong Liu, Zerun Ma, Zixing Shang, Wenhui Tian, Zun Wang, Liwei Wu, Zhenyu Wu, Jun Xu, Bowen Yang, Dingbo Yuan, Qi Zhang, Songyang Zhang, Peiheng Zhou, Dongsheng Zhu

llms agents benchmarks

arXiv Machine Learning

Jul 16

Text2Sign: A Single-GPU Diffusion Baseline for Text-to-Sign Language Video Generation

arXiv:2607. 13164v1 Announce Type: cross Abstract: Sign language is a primary communication channel for millions of Deaf and hard-of-hearing people, yet text-to-signer video generation remains costly because video diffusion models are expensive to train and evaluate.

By Ruize Xia

llms diffusion multimodal benchmarks

arXiv AI

Jul 16

Federated Explainable Artificial Intelligence: Roles, Architectures, Evaluation, and Open Challenges

arXiv:2607. 13045v1 Announce Type: cross Abstract: Federated Learning (FL) has emerged as a key paradigm for privacy-preserving collaborative model training across distributed and heterogeneous data sources.

By Masoume Gholizade, Fabrizio Ruffini, Pietro Ducange, Francesco Marcelloni

benchmarks safety

arXiv AI

Jul 16

Stable Attention Response for Reliable Precipitation Nowcasting

arXiv:2605. 13181v2 Announce Type: replace-cross Abstract: Precipitation nowcasting remains challenging due to the highly localized, rapidly evolving, and heterogeneous nature of atmospheric dynamics.

By Penghui Wen, Zexin Hu, Sen Zhang, Patrick Filippi, Xiaogang Zhu, Allen Benter, Thomas Bishop, Zhiyong Wang, Kun Hu

diffusion multimodal benchmarks

arXiv AI

Jul 16

Beyond Backbone Backpropagation: A Decoupled Strategy for Efficient Transfer Learning

arXiv:2607. 13043v1 Announce Type: cross Abstract: Deep learning models achieve state-of-the-art image classification but face deployment challenges due to computational costs and energy demands.

By Daniel Vila-Cruz, Laura Mor\'an-Fern\'andez, Ver\'onica Bol\'on-Canedo

llms computer-vision fine-tuning benchmarks

arXiv Machine Learning

Jul 16

Leveraging Differentiable PDE Solvers for Semi-Neural Spatial Reconstruction From Sparse Measurements

arXiv:2601. 20496v2 Announce Type: replace-cross Abstract: Generating dense physical fields from sparse measurements is a fundamental question in sampling, signal processing, and many other applications.

By Ofek Aloni, Barak Fishbain

benchmarks

arXiv AI

Jul 16

EMAGN: Efficient Multi-Attention Graph Network via Learned Clustering for Scalable Traffic Forecasting

arXiv:2607. 13241v1 Announce Type: cross Abstract: Traffic forecasting is highly challenging due to complex and nonlinear spatial and temporal dependencies.

By Mingxing Xu, Rakesh Chowdary Machineni, Ke Liu, Xi Cheng, Chengqi Lu, Xin Hu, Lyuhao Chen, Xiangyu Li, Junwei You, Oliver Gao

benchmarks

arXiv AI

Jul 16

Discovering Ordinary Differential Equations with LLM-Based Qualitative and Quantitative Evaluation

arXiv:2605. 07323v2 Announce Type: replace Abstract: Discovering governing differential equations from observational data is a fundamental challenge in scientific machine learning.

By Sum Kyun Song, Bong Gyun Shin, Jae Yong Lee

llms agents benchmarks

arXiv Machine Learning

Jul 16

BARS: Benign-Anchored Ranking and Selection for False Alarm Reduction in Network Intrusion Detection

arXiv:2607. 13203v1 Announce Type: cross Abstract: False alarms remain a major barrier to deploying network intrusion detection systems (NIDS).

By Abu Fuad Ahmad, Istiaque Ahmed

efficiency benchmarks safety

arXiv Machine Learning

Jul 16

BenthiCat: An opti-acoustic dataset for advancing benthic classification and habitat mapping

arXiv:2510. 04876v3 Announce Type: replace-cross Abstract: Benthic habitat mapping is fundamental for understanding marine ecosystems, guiding conservation efforts, and supporting sustainable resource management.

By Hayat Rajani, Valerio Franchi, Borja Martinez-Clavel Valles, Raimon Ramos, Rafael Garcia, Nuno Gracias

agents computer-vision fine-tuning multimodal benchmarks

Get the daily briefing