Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

23,273 stories · RSS feed

arXiv Machine Learning

Jun 8

Learning Explicit Behavioral Models with Adaptive Questions and World-Model Probes

arXiv:2606. 07127v1 Announce Type: new Abstract: Interactive agents trained only against task return can achieve high scores while failing to represent the mechanisms that make their actions succeed.

By Hikaru Shindo, Yu Deng, Teng Cao, Quentin Delfosse, Christopher Tauchmann, Jannis Bl\"uml, Gopika Sudhakaran, Kristian Kersting

llms agents nlp benchmarks

arXiv Machine Learning

Jun 8

Closed-Form Spectral Regularization for Multi-Task Model Merging

arXiv:2606. 07289v1 Announce Type: new Abstract: Model merging combines several independently fine-tuned experts into a single multi-task model without any training data, reducing the storage, serving, and decentralized-development costs of large foundation models.

By Yongxian Wei, Runxi Cheng, Xingxuan Zhang, Li Shen, Chun Yuan, Peng Cui, Dacheng Tao

computer-vision fine-tuning multimodal benchmarks

arXiv Machine Learning

Jun 8

Accelerating Reproducible Research in Synthetic EHR Generation

arXiv:2606. 06990v1 Announce Type: new Abstract: The generation of high-fidelity synthetic Electronic Health Records (EHR) is crucial for advancing medical research while preserving patient privacy.

By Jalen Jiang, Chufan Gao, Ethan Rasmussen, Stephen Z. Xie, Jimeng Sun

llms diffusion benchmarks safety

arXiv AI

Jun 8

Standard vs. Modular Sampling: Best Practices for Reliable LLM Unlearning

arXiv:2509. 05316v2 Announce Type: replace-cross Abstract: A conventional LLM Unlearning setting consists of two subsets -"forget" and "retain", with the objectives of removing the undesired knowledge from the forget set while preserving the remaining knowledge from the retain.

By Praveen Bushipaka, Lucia Passaro, Tommaso Cucinotta

llms benchmarks safety

arXiv AI

Jun 8

ViVa: A Video-Generative Value Model for Robot Reinforcement Learning

arXiv:2604. 08168v2 Announce Type: replace-cross Abstract: Vision-language-action (VLA) models have advanced robot manipulation through large-scale pretraining, but real-world deployment remains challenging due to partial observability and delayed feedback.

By Jindi Lv, Hao Li, Jie Li, Fankun Kong, Yang Wang, Pengfei Yi, Yifei Nie, Xiaofeng Wang, Zheng Zhu, Chaojun Ni, Qiuping Deng, Hengtao Li, Jiancheng Lv, Guan Huang

llms reinforcement-learning robotics multimodal benchmarks

arXiv AI

Jun 8

CoQuIR: A Comprehensive Benchmark for Code Quality-Aware Information Retrieval

arXiv:2506. 11066v3 Announce Type: replace-cross Abstract: Code retrieval is essential in modern software development, as it boosts code reuse and accelerates debugging.

By Jiahui Geng, Fengyu Cai, Shaobo Cui, Qing Li, Liangwei Chen, Chenyang Lyu, Haonan Li, Derui Zhu, Walter Pretschner, Heinz Koeppl, Fakhri Karray

benchmarks

arXiv Machine Learning

Jun 8

$\alpha$-PFN: Fast Entropy Search via In-Context Learning

arXiv:2606. 07134v1 Announce Type: new Abstract: Information-theoretic acquisition functions such as Entropy Search (ES) offer a principled exploration-exploitation framework for Bayesian optimization (BO).

By Herilalaina Rakotoarison, Steven Adriaensen, Tom Viering, Carl Hvarfner, Samuel M\"uller, Frank Hutter, Eytan Bakshy

llms benchmarks

arXiv AI

Jun 8

MoDA: Modulation Adapter for Fine-Grained Visual Grounding in Instructional MLLMs

arXiv:2506. 01850v2 Announce Type: replace-cross Abstract: Multimodal Large Language Models (MLLMs) have achieved remarkable success in instruction-following tasks by integrating pretrained visual encoders with large language models (LLMs).

By Wayner Barrios, Andr\'es Villa, Juan Le\'on Alc\'azar, SouYoung Jin, Bernard Ghanem

llms rag nlp multimodal benchmarks safety

arXiv AI

Jun 8

Towards Efficient and Exact Forgetting Services in Pre-Trained-Model-based Continual Learning

arXiv:2505. 12239v2 Announce Type: replace-cross Abstract: In Continual Learning (CL), using a Pre-Trained Model (PTM) as the feature extractor has become a popular practice.

By Yajiang Huang, Jianheng Tang, Kejia Fan, Huiping Zhuang, Anfeng Liu, Tian Wang, Yunhuai Liu, Mianxiong Dong, Houbing Herbert Song

benchmarks safety

arXiv AI

Jun 8

Beyond the Black Box: Interpretability of Agentic AI Tool Use

arXiv:2605. 06890v3 Announce Type: replace Abstract: AI agents are promising for high-stakes enterprise workflows, but dependable deployment remains limited because tool-use failures are difficult to diagnose and control.

By Hariom Tatsat, Ariye Shater

llms agents benchmarks safety

arXiv AI

Jun 8

CrowdMath: A Dataset of Crowdsourced Mathematical Research Discussions

arXiv:2606. 06526v1 Announce Type: new Abstract: Large language models have made substantial progress on mathematical reasoning, but existing benchmarks typically evaluate well-specified problems with final answers, step-by-step solutions, or complete proofs.

By Sherin Muckatira, Jesse Geneson, Slava Gerovitch, Pavel Etingof, Mikhail Gronas, Anna Rumshisky

llms benchmarks

arXiv AI

Jun 8

Graph Neural Network leveraging Higher-order Class Label Connectivity for Heterophilous Graphs

arXiv:2606. 07475v1 Announce Type: cross Abstract: Node classification in graph neural networks (GNNs) has been widely applied in various fields of graph analysis.

By Takuto Takahashi, Itsuki Nakayama, Takahiro Mitani, Ryosuke Kikuchi, Yuya Sasaki, Makoto Onizuka

rag benchmarks

arXiv Machine Learning

Jun 8

Generative Molecular Morphing for Flexible-Size Design via Unbalanced Optimal Transport

arXiv:2606. 07239v1 Announce Type: new Abstract: The success of generative molecular design hinges on a model's steerability toward high-reward samples.

By Malte Franke, Stefan P. Schmid, Zarko Ivkovic, Kjell Jorner, Andreas Krause

diffusion benchmarks

arXiv AI

Jun 8

COF26: A new on-top functional for multiconfiguration pair-density functional theory

arXiv:2605. 06215v2 Announce Type: replace-cross Abstract: Multiconfiguration pair-density functional theory (MC-PDFT) provides an efficient and accurate framework for computing electronic energies in strongly correlated molecular systems, with the quality of the on-top functional being a key determinant of its predictive accuracy.

By Yuhao Chen, Donald G. Truhlar, Xiao He

benchmarks

arXiv Machine Learning

Jun 8

Uncertainty-Aware LLM-Guided Policy Shaping for Sparse-Reward Reinforcement Learning

arXiv:2606. 06673v1 Announce Type: new Abstract: Sparse rewards and heterogeneous task sequences remain persistent challenges in Reinforcement Learning (RL), often resulting in slow convergence, weak generalization, and inefficient exploration.

By Ujjwal Bhatta, Utsabi Dangol, Sumaly Bajracharya, Rodrigue Rizk, KC Santosh

llms agents reinforcement-learning fine-tuning benchmarks

arXiv AI

Jun 8

Scalable GANs with Transformers

arXiv:2509. 24935v3 Announce Type: replace-cross Abstract: Scalability has driven recent advances in generative modeling, yet its principles remain underexplored for adversarial learning.

By Sangeek Hyun, MinKyu Lee, Jae-Pil Heo

llms diffusion benchmarks safety

arXiv AI

Jun 8

Focus-then-Context: Subject-Centric Progressive Visual Token Reduction for Vision-Language Models

arXiv:2605. 20950v2 Announce Type: replace-cross Abstract: Vision-Language Models (VLMs) face a bottleneck of prohibitive computational costs arising from massive visual token sequences during inference.

By Yulin Zhao, Zheng Zhang

llms multimodal benchmarks

arXiv AI

Jun 8

Beyond Skeletons: Learning Animation Directly from Driving Videos with Same2X Training Strategy

arXiv:2606. 06903v1 Announce Type: cross Abstract: Human image animation aims to generate a video from a static reference image, guided by pose information extracted from a driving video.

By Yuan Zeng, Yujia Shi, Yuhao Yang, Dongxia Liu, Zongqing Lu, Wenming Yang, Qingmin Liao

benchmarks safety

arXiv AI

Jun 8

Hierarchical Certified Semantic Commitment for Byzantine-Resilient LLM-Agent Collaboration

arXiv:2606. 07316v1 Announce Type: cross Abstract: Byzantine collaboration among large-language-model agents requires a finality-control primitive: given delivered stochastic, structured natural-language proposals, the protocol must decide whether the round supports a commit, what kind of commit, or a typed safe abort.

By Haoran Xu, Lei Zhang, Iadh Ounis, Xianbin Wang

llms rag agents benchmarks safety

arXiv AI

Jun 8

REMEDI: A Benchmark for Retention and Unlearning Evaluation in Multi-label Clinical Disease Inference

arXiv:2606. 07141v1 Announce Type: cross Abstract: Language models trained for clinical disease inference are trained on patient data, which may include sensitive and private information, and data owners may request the removal of their data from a trained model due to privacy or copyright concerns.

By Anurag Sharma, Sai Teja Chunchu, Prasenjit Mitra, Sandipan Sikdar, Koustav Rudra

llms benchmarks safety

Get the daily briefing