Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

18,930 stories · RSS feed

arXiv Machine Learning

Jun 30

Shoot from the HIP: Hessian Interatomic Potentials without derivatives

arXiv:2509. 21624v3 Announce Type: replace Abstract: Fundamental tasks in computational chemistry, from transition state search to vibrational analysis, rely on molecular Hessians, which are the second derivatives of the potential energy.

By Andreas Burger, Luca Thiede, Nikolaj R{\o}nne, Varinia Bernales, Nandita Vijaykumar, Tejs Vegge, Arghya Bhowmik, Alan Aspuru-Guzik

benchmarks

arXiv AI

Jun 30

TERC: A Transfer Entropy Redundancy Criterion for State Variable Selection in Reinforcement Learning

arXiv:2401. 11512v2 Announce Type: replace-cross Abstract: Identifying the most suitable variables to represent the state is a fundamental challenge in Reinforcement Learning (RL).

By Charles Westphal, Stephen Hailes, Mirco Musolesi

agents reinforcement-learning benchmarks

arXiv AI

Jun 30

DLR: Zero-Inference-Cost Latent Residuals for Low-Rank Pre-Training

arXiv:2606. 28932v1 Announce Type: cross Abstract: Large language models have driven recent progress in language and multimodal AI, yet pre-training them at scale is prohibitively expensive.

By Dong Wang, Wenwu Tang, Yun Cheng, Olga Saukh

llms fine-tuning multimodal benchmarks

arXiv AI

Jun 30

SFBench: The SciFy Scientific Feasibility Benchmark

arXiv:2606. 29630v1 Announce Type: new Abstract: We present SFBench, a benchmark dataset for evaluating systems that assess the feasibility of scientific claims.

By Cash Costello, James Mayfield, Elsbeth Turcan, Christine Piatko, Christina K. Pikas, Justin Rokisky, Sam Scheck, Chris Ribaudo, Ritwik Bose, Alex Memory

llms benchmarks

arXiv AI

Jun 30

MedCase-Structured: A Text-to-FHIR Dataset for Benchmarking Diagnostic Reasoning in Clinically Realistic EHR Settings

arXiv:2605. 30295v2 Announce Type: replace-cross Abstract: Large language models (LLMs) show promise for clinical reasoning and decision support, but evaluation in realistic, electronic health record-congruent settings remains limited.

By Valentina Bui Muti, Eug\'enie Dulout, Ziquan Fu

llms benchmarks

arXiv AI

Jun 30

Towards Continual Motion-Language Agents: LoRA Variants for Incremental Motion Understanding and Generation

arXiv:2606. 30266v1 Announce Type: cross Abstract: Motion-language agents must possess the bidirectional capability to both understand human movement (motion-to-text, M2T) and generate it from natural language (text-to-motion, T2M).

By Bertram Taetz, Hugo Albuquerque Cosme da Silva, Gabriele Bleser-Taetz

llms agents fine-tuning benchmarks

arXiv Machine Learning

Jun 30

SWE-INTERACT: Reimagining SWE Benchmarks as User-Driven Long-Horizon Coding Sessions

arXiv:2606. 30573v1 Announce Type: new Abstract: We introduce SWE-Interact, a new testbed for evaluating coding agents on multi-turn, interactive, user-driven software engineering tasks.

By Mohit Raghavendra, Anisha Gunjal, Aakash Sabharwal, Yunzhong He

llms agents benchmarks

arXiv AI

Jun 30

CMTFormer: Marrying Transformer with Hierarchical Information Interaction for RGB-Event Object Detection

arXiv:2606. 29136v1 Announce Type: cross Abstract: Event cameras capture sparse brightness changes with high temporal resolution and high dynamic range, compensating for the deficiencies of the conventional RGB frames.

By Yu Li, Yuenan Hou, Yingmei Wei, Jiangming Chen, Yanming Guo

llms computer-vision multimodal benchmarks safety

arXiv AI

Jun 30

Beyond Point Estimates for Glaucoma Visual Field Forecasting with Diffusion Models

arXiv:2606. 30417v1 Announce Type: cross Abstract: Forecasting visual fields (VFs) is critical for personalized monitoring and treatment planning in glaucoma.

By Marta Colmenar Herrera, Pablo M\'arquez Neila, \c{S}erife Seda Kucur Erg\"unay, Martin S. Zinkernagel, Raphael Sznitman

diffusion benchmarks

arXiv AI

Jun 30

Counterfactual Residual Data Augmentation for Regression

arXiv:2606. 28460v1 Announce Type: cross Abstract: Data-driven modeling in real-world regression tasks often suffers from limited training samples, high collection costs, and noisy observations.

By Hossein Mohebbi, Oliver Schulte, Ke Li, Pascal Poupart

benchmarks

arXiv AI

Jun 30

Code Reasoning for Software Engineering Tasks: A Survey and A Call to Action

arXiv:2506. 13932v3 Announce Type: replace-cross Abstract: The rise of large language models (LLMs) has led to dramatic improvements across a wide range of natural language tasks.

By Saurabh Pujar, Ira Ceka, Irene Manotas, Gail Kaiser, Baishakhi Ray, Shyam Ramji

llms agents benchmarks

arXiv AI

Jun 30

Can Fine-Tuning Erase Your Edits? On the Fragile Coexistence of Knowledge Editing and Adaptation

arXiv:2511. 05852v4 Announce Type: replace-cross Abstract: Knowledge editing (KE) offers a lightweight alternative to retraining for updating large language models (LLMs).

By Yinjie Cheng, Paul Youssef, Christin Seifert, J\"org Schl\"otterer, Zhixue Zhao

llms fine-tuning benchmarks safety

arXiv AI

Jun 30

Edit in 2D, Verify in 3D: Reinforcement Learning for Multi-view Consistent Scene Editing

arXiv:2603. 03143v2 Announce Type: replace-cross Abstract: Leveraging the priors of 2D diffusion models for 3D editing has emerged as a promising paradigm.

By Jiyuan Wang, Chunyu Lin, Lei Sun, Zhi Cao, Yuyang Yin, Lang Nie, Zhenlong Yuan, Xiangxiang Chu, Yunchao Wei, Kang Liao, Guosheng Lin

diffusion computer-vision reinforcement-learning fine-tuning benchmarks

arXiv AI

Jun 30

BTI-Net: Bidirectional Decoder-Level Task Interaction via Uncertainty-Aware Gating for Multi-Task Medical Image Analysis

arXiv:2606. 29102v1 Announce Type: cross Abstract: Jointly learning to segment and classify medical images demands cross-task synergy, yet encoder-sharing architectures limit decoder reconstruction to task-private representations, permanently discarding the boundary cues and semantic priors each branch could supply to the other.

By Abdullah Al Shafi, Md Kawsar Mahmud Khan Zunayed, Safin Ahmmed, Sk Imran Hossain, Engelbert Mephu Nguifo

computer-vision benchmarks safety

arXiv AI

Jun 30

Grounding Sim-to-Real Generalization in Robotic Manipulation: An Empirical Study with Vision-Language-Action Models

arXiv:2603. 22876v2 Announce Type: replace-cross Abstract: Learning a generalist control policy for robotic manipulation typically relies on large-scale datasets.

By Ruixing Jin, Zicheng Zhu, Ruixiang Ouyang, Sheng Xu, Bo Yue, Zhizheng Wu, Guiliang Liu

reinforcement-learning robotics multimodal benchmarks

arXiv Machine Learning

Jun 30

Beyond Trajectory Matching: Reflow with Marginal Distribution Alignment

arXiv:2606. 29287v1 Announce Type: new Abstract: Diffusion and continuous-flow generative models achieve high-quality generation, and their deterministic sampling can be formulated as solving learned ODE dynamics.

By Chen Wang, Peiran Yun, Pan Xie, Ke Deng

diffusion efficiency benchmarks safety

arXiv AI

Jun 30

Clinical Reasoning Graphs: Structured Evaluation of LLM Diagnostic Reasoning Reveals Competence Without Consistency

arXiv:2606. 29876v1 Announce Type: cross Abstract: Modern large language models (LLMs) reach 60-70% diagnostic accuracy on complex clinical case benchmarks, but accuracy alone cannot distinguish stable clinically-grounded reasoning from pattern matching.

By Nisarg A. Patel (University of California, San Francisco)

llms benchmarks

arXiv AI

Jun 30

A Multi-Dataset Benchmark for Evaluating LLM Agents in Microservice Failure Diagnosis

arXiv:2606. 29193v1 Announce Type: cross Abstract: LLM-based agents are reshaping microservice operations into AgentOps, where benchmarks are key to evaluating failure diagnosis over multimodal observability data.

By Yuanhong Cai, Xiaohui Nie, Kanglin Yin, Changhua Pei, Yongqian Sun, Shenglin Zhang, Haibin Liu, Guiyang Liu, Xidao Wen, Fang Situ, Dan Pei

llms agents multimodal benchmarks

arXiv AI

Jun 30

Diversity is the Strength of the AI Crowd

arXiv:2606. 29661v1 Announce Type: new Abstract: Top AI forecasting systems are approaching superforecaster-level accuracy on future world events, but still rely primarily on off-the-shelf LLMs combined with forecasting-specific context gathering and scaffolding.

By Matthew Aitchison, Scott Jeen, Toby Shevlane, Ben Day

llms benchmarks

arXiv Machine Learning

Jun 30

CoLR-Det: Collaborative Latent Restoration for Small Object Detection in Low-Resolution Remote Sensing Images

arXiv:2601. 12507v2 Announce Type: replace-cross Abstract: Low-resolution remote sensing small object detection is limited by both missing visual details and the ambiguity of how details serve detection.

By Ruo Qi, Linhui Dai, Yusong Qin, Chaolei Yang, Yanshan Li

computer-vision benchmarks

Get the daily briefing