Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

22,941 stories · RSS feed

arXiv AI

Jun 9

Visual Prompting Meets Feature Reconstruction-Based Anomaly Detection with Dual-Teacher Supervision

arXiv:2606. 09670v1 Announce Type: cross Abstract: Recent Anomaly Detection methods achieve perfect detection and segmentation scores on well-established datasets, such as MVTec.

By Mateo Diaz-Bone, Daniel Caraballo, Florian Scheidegger, Thomas Frick, Mattia Rigotti, Andrea Bartezzaghi, Roy Assaf, Niccolo Avogaro, Yagmur G. Cinar, Brown Ebouky, Filip M. Janicki, Piotr S. Kluska, Cezary Skura, Cristiano Malossi

llms diffusion computer-vision benchmarks

arXiv AI

Jun 9

RTL-BenchLS: A Large-Scale Benchmark for RTL Reasoning and Generation with Large Language Models

arXiv:2606. 08976v1 Announce Type: new Abstract: LLM-based RTL generation and reasoning is a promising direction for hardware design automation.

By Jing Wang, Shang Liu, Wenji Fang, Yuchao Wu, Yugao Zhu, Zhiyao Xie

llms benchmarks

arXiv AI

Jun 9

LoTUS: Large-Scale Machine Unlearning with a Taste of Uncertainty

arXiv:2503. 18314v5 Announce Type: replace-cross Abstract: We present LoTUS, a novel Machine Unlearning (MU) method that eliminates the influence of training samples from pre-trained models, avoiding retraining from scratch.

By Christoforos N. Spartalis, Theodoros Semertzidis, Petros Daras, Efstratios Gavves

llms benchmarks

arXiv AI

Jun 9

TABVERSE: Benchmarking Cross-Format Table Understanding in LLMs and VLMs

arXiv:2606. 09578v1 Announce Type: new Abstract: Large Language Models (LLMs) and Vision-Language Models (VLMs) are increasingly evaluated on table reasoning tasks, but the role of table representation remains under-explored.

By Momina Ahsan, Sarfraz Ahmad, Ming Shan Hee, Roy Ka-Wei Lee, Preslav Nakov

llms nlp multimodal benchmarks

arXiv AI

Jun 9

RunAgent SuperBrowser: A Theory of Autonomous Web Navigation Grounded in Human Browsing Behaviour

arXiv:2606. 09399v1 Announce Type: new Abstract: We present SUPERBROWSER, an autonomous web-navigation agent designed against a single guiding hypothesis: a web agent should browse the way a person browses.

By Radeen Mostafa, Sawradip Saha

llms agents benchmarks

arXiv Machine Learning

Jun 9

A Comparison of SSL-Based Feature Extractors and Back-End Classifiers for Spoofing Detection: A Multi-Corpus Training and Cross-Linguistic Analysis

arXiv:2606. 08669v1 Announce Type: cross Abstract: Voice biometric systems face growing threats from spoofing attacks, yet the evaluation of detection models remains inconsistent across datasets.

By Anh-Tuan Dao, Driss Matrouf, Mickael Rouvier, Nicholas Evans

fine-tuning benchmarks safety

arXiv AI

Jun 9

A Regret Minimization Framework on Preference Learning in Large Language Models

arXiv:2606. 09124v1 Announce Type: new Abstract: Reinforcement learning with verifiable rewards (RLVR) has enabled progress on reasoning-intensive tasks by relying on task-specific verifiers that provide automated correctness signals.

By Suhwan Kim, Taehyun Cho, Geon-Hyeong Kim, Yu Jin Kim, Youngsoo Jang, Moontae Lee, Jungwoo Lee

llms reinforcement-learning benchmarks

arXiv AI

Jun 9

Training-Inference Kernel Contracts: Bounding Divergence in Post-Training and Deployment

arXiv:2606. 07581v1 Announce Type: cross Abstract: A modern post-training pipeline often writes one symbol for its policy, pi_theta, while evaluating it through two different programs: a training kernel optimized for autograd and an inference kernel optimized for low-precision, fused, dynamically batched serving.

By Bruce Changlong Xu, Lan Wu

reinforcement-learning benchmarks safety

arXiv AI

Jun 9

Pharmacogenomic Knowledge Graph Augmentation for Graph Neural Network-Based Drug-Drug Interaction Prediction

arXiv:2606. 07698v1 Announce Type: cross Abstract: Graph neural networks (GNNs) applied to drug-drug interaction (DDI) prediction rely exclusively on molecular structure encoded as SMILES-derived graphs.

By Juergen Dietrich

rag multimodal benchmarks safety

arXiv Machine Learning

Jun 9

TinyJudge: Unverifiable Constraint Alignment via Lightweight Specialist Ensembles

arXiv:2606. 07520v1 Announce Type: cross Abstract: Instruction Following (IF) is a core capability of LLMs, requiring strict adherence to diverse constraints, ranging from verifiable ones (e.

By Yirong Zeng, Yufei Liu, Xiao Ding, Yutai Hou, Yuxian Wang, Wu Ning, Haonan Song, Dandan Tu, Qixun Zhang, Yuxiang He, Bibo Cai, Ting Liu

llms reinforcement-learning benchmarks safety

arXiv Machine Learning

Jun 9

AccioScene: Compositional 3D Scene Generation via Graph Diffusion and Interaction-driven Critics

arXiv:2502. 06819v2 Announce Type: replace Abstract: This paper presents a framework for generating 3D indoor scenes from text prompts.

By Yao Wei, Matteo Toso, Pietro Morerio, Changjae Oh, Michael Ying Yang, Alessio Del Bue

diffusion benchmarks

arXiv AI

Jun 9

NutriMLLM: Multimodal Large Language Models for Dietary Micronutrient Analysis

arXiv:2606. 08948v1 Announce Type: cross Abstract: Comprehensive estimation of dietary micronutrients from food images could improve clinical nutrition care, but training such models requires large multimodal datasets linking diverse foods to complete nutrient profiles.

By Runze Yan, Minxiao Wang, Jiaying Lu, Darren Liu, Xiao Hu, Hanqi Luo

llms diffusion fine-tuning multimodal benchmarks safety

arXiv AI

Jun 9

When Behavioral Safety Evaluation Fails: A Representation-Level Perspective

arXiv:2606. 08044v1 Announce Type: cross Abstract: Large Language Model (LLM) safety has often been evaluated at the behavior level, which provides limited evidence of internal robustness, as these evaluations target outputs rather than representation-level vulnerability under intervention.

By Enyi Jiang, Anders Gj{\o}lbye, Yibo Jacky Zhang, Sanmi Koyejo

llms diffusion fine-tuning benchmarks safety

arXiv AI

Jun 9

SafeRun: Enabling Determinism in LLM Planning for Running

arXiv:2606. 09027v1 Announce Type: cross Abstract: Large Language Models enable flexible natural-language planning but remain unreliable in determinism-critical domains due to their probabilistic nature.

By Meilin Chen, Zepeng Zhai, Jiaxuan Zhao, Yuan Lu

llms benchmarks safety

arXiv Machine Learning

Jun 9

Variational Proximal Policy Optimization

arXiv:2606. 08032v1 Announce Type: cross Abstract: Reinforcement Learning from Human Feedback via Proximal Policy Optimization often suffers from policy mode collapse, brittle exploration loops, and distribution drift.

By Ousmane Amadou Dia

reinforcement-learning benchmarks

arXiv AI

Jun 9

Proposal Refinement for Few-Shot Object Detection

arXiv:2606. 09245v1 Announce Type: cross Abstract: Few-shot object detection has gained widely attention in recent years.

By Yuan Zeng, Bin Song, Jie Guo, Yuwen Chen

computer-vision fine-tuning benchmarks

arXiv AI

Jun 9

GraphLoRA: Structure-Aware Low-Rank Adaptation for Large Language Model Recommendation

arXiv:2606. 07526v1 Announce Type: cross Abstract: Large Language Models (LLMs) have shown strong potential for recommendation (LLMRec) due to their powerful reasoning and generalization abilities.

By Lin Mu, Guoji Wang, Li Ni, Lei Sang, Zhize Wu, Peiquan Jin, Yiwen Zhang

llms rag benchmarks

arXiv Machine Learning

Jun 9

In-Context Learning of Temporal Point Processes with Foundation Inference Models

arXiv:2509. 24762v3 Announce Type: replace Abstract: Modeling event sequences of multiple event types with marked temporal point processes (MTPPs) provides a principled way to uncover governing dynamical rules and predict future events.

By David Berghaus, Patrick Seifner, Kostadin Cvejoski, C\'esar Ojeda, Rams\'es J. S\'anchez

llms benchmarks

arXiv Machine Learning

Jun 9

Modeling Stochastic Conditional Dynamics from Sparse Observations via Kernel-Stabilized Flow Matching

arXiv:2411. 08314v5 Announce Type: replace Abstract: Learning to transform conditional probability densities over time is a fundamental challenge spanning probabilistic modeling and the natural sciences.

By Adam P. Generale, Andreas E. Robertson, Surya R. Kalidindi

diffusion benchmarks

arXiv AI

Jun 9

Correct Looks Better: Pairwise Comparisons Reveal Accuracy Rankings

arXiv:2606. 09409v1 Announce Type: new Abstract: Pairwise comparisons combined with aggregation methods like Elo have become central to evaluating generative models, yet concerns remain that they reward superficial stylistic cues or display judge biases.

By Mina Remeli, Moritz Hardt

benchmarks safety

Get the daily briefing