Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

24,477 stories · RSS feed

arXiv AI

Jun 4

Spectral Scaling Laws of Muon

arXiv:2606. 04058v1 Announce Type: cross Abstract: Orthonormalized update rules have rapidly become a leading choice of optimizer for training large language models, with recent open-source state-of-the-art models adopting Muon.

By Gagik Magakyan, Pablo Parrilo, Asuman Ozdaglar

llms benchmarks

arXiv Machine Learning

Jun 4

Outcome-Grounded Advantage Reshaping for Fine-Grained Credit Assignment in Mathematical Reasoning

arXiv:2601. 07408v2 Announce Type: replace-cross Abstract: Group Relative Policy Optimization (GRPO) has emerged as a promising critic-free reinforcement learning paradigm for reasoning tasks.

By Ziheng Li, Liu Kang, Feng Xiao, Luxi Xing, Qingyi Si, Zhuoran Li, Weikang Gong, Deqing Yang, Yanghua Xiao, Hongcheng Guo

llms reinforcement-learning benchmarks

arXiv Machine Learning

Jun 4

Plug-and-Play Diffusion Meets ADMM: Dual-Variable Coupling for Robust Medical Image Reconstruction

arXiv:2602. 23214v2 Announce Type: replace-cross Abstract: Plug-and-Play diffusion prior (PnPDP) frameworks have emerged as a powerful paradigm for solving imaging inverse problems by treating pretrained generative models as modular priors.

By Chenhe Du, Xuanyu Tian, Qing Wu, Muyu Liu, Jingyi Yu, Hongjiang Wei, Yuyao Zhang

diffusion fine-tuning benchmarks safety

arXiv Machine Learning

Jun 4

Efficient and Training-Free Single-Image Diffusion Models

arXiv:2606. 04299v1 Announce Type: cross Abstract: We consider the problem of generating images whose internal structure -- defined by the distribution of patches across multiple scales -- matches that of a single reference image.

By Haojun Qiu, Kiriakos N. Kutulakos, David B. Lindell

diffusion benchmarks

arXiv AI

Jun 4

AICompanionBench: Benchmarking LLMs-as-Judges for AI Companion Safety

arXiv:2606. 04867v1 Announce Type: new Abstract: As AI companion platforms such as Replika and Character.

By Yanjing Ren, Reza Ebrahimi, TengTeng Ma

llms robotics benchmarks safety

arXiv AI

Jun 4

DiffAero: A GPU-Accelerated Differentiable Simulation Framework for Efficient Quadrotor Policy Learning

arXiv:2509. 10247v1 Announce Type: cross Abstract: This letter introduces DiffAero, a lightweight, GPU-accelerated, and fully differentiable simulation framework designed for efficient quadrotor control policy learning.

By Xinhong Zhang, Runqing Wang, Yunfan Ren, Jian Sun, Hao Fang, Jie Chen, Gang Wang

agents efficiency benchmarks

arXiv Machine Learning

Jun 4

Attention-Based Sampler for Diffusion Language Models

arXiv:2604. 08564v2 Announce Type: replace-cross Abstract: Auto-regressive models (ARMs) have established a dominant paradigm in language modeling.

By Yuyan Zhou, Kai Syun Hou, Weiyu Chen, James Kwok

llms diffusion benchmarks

arXiv Machine Learning

Jun 4

Neural Autoregressive Control Variates for the Quantum Monte Carlo Sign Problem

arXiv:2605. 26814v2 Announce Type: replace-cross Abstract: We train a pair of autoregressive models to construct zero-mean control variates to mitigate the sign problem in quantum Monte Carlo simulations.

By Bei Qiao, Lei Wang

llms benchmarks

arXiv Machine Learning

Jun 4

CADET: A Modular Platform for Evaluating Distributed Cooperative Autonomy in Connected Autonomous Vehicles

arXiv:2606. 04072v1 Announce Type: cross Abstract: Deep learning models are increasingly central to autonomous vehicle (AV) pipelines, yet their integration has traditionally followed a monolithic design where perception, planning, and control execute on a single onboard computer.

By Pragya Sharma, Brian Wang, Mani Srivastava

agents benchmarks safety

arXiv Machine Learning

Jun 4

Bayesian Membership Privacy for Graph Neural Networks

arXiv:2606. 04069v1 Announce Type: cross Abstract: Existing privacy analyses for Graph Neural Networks (GNNs) largely inherit assumptions from non-graph settings, overlooking structural correlations and stochastic training-graph sampling.

By Sinan Y{\i}ld{\i}r{\i}m, Megha Khosla

benchmarks safety

arXiv AI

Jun 4

BiasGRPO: Stabilizing Bias Mitigation in High-Variance Reward Landscapes via Group-Relative Policy Optimization

arXiv:2606. 04807v1 Announce Type: new Abstract: Mitigating social bias in Large Language Models (LLMs) presents a distinct alignment challenge: unlike verifiable tasks, bias lacks a single ground truth, creating a high-variance, subjective reward landscape.

By Saket Reddy, Ke Yang, ChengXiang Zhai

llms reinforcement-learning fine-tuning benchmarks safety

arXiv Machine Learning

Jun 4

STRIDE: Training Data Attribution via Sparse Recovery from Subset Perturbations

arXiv:2606. 05165v1 Announce Type: new Abstract: Training Data Attribution (TDA) seeks to trace a model's predictions back to its training data.

By Rishit Dagli, Abir Harrasse, Luke Zhang, Florent Draye, Amirali Abdullah, Bernhard Sch\"olkopf, Zhijing Jin

llms benchmarks

arXiv Machine Learning

Jun 4

RePercENT: Scaling Disentangled Representation Learning Beyond Two Modalities

arXiv:2606. 05109v1 Announce Type: new Abstract: To leverage the full potential of multimodal data, we need representations that go beyond the state-of-the-art alignment and fusion approaches and exploit all cross-modal interactions without sacrificing modality-specific information.

By Vasiliki Rizou, Pascal Frossard, Dorina Thanou

rag multimodal benchmarks safety

arXiv Machine Learning

Jun 4

In-Context Graphical Inference

arXiv:2606. 05042v1 Announce Type: new Abstract: Marginal inference in discrete graphical models forces a choice between exactness and scalability: exact algorithms are intractable for high-treewidth graphs, while iterative approximations (Belief Propagation, variational methods) sacrifice convergence guarantees on frustrated topologies.

By Zehua Cheng, Wei Dai, Jiahao Sun

llms benchmarks safety

arXiv Machine Learning

Jun 4

Rethinking Incompleteness: Formalizing Protocol Divergence and Train-Once Learning for Robust IMVC

arXiv:2606. 04857v1 Announce Type: new Abstract: Standard IMVC evaluation retrains separate models for different missing-data configurations.

By Haolu Liu, Xiyue Wang, Xuanting Xie, Liangjian Wen, Zhao Kang

llms benchmarks

arXiv Machine Learning

Jun 4

ALINC: Active Learning for Inductive Node Classification via Graph Sampling

arXiv:2606. 04647v1 Announce Type: new Abstract: Active learning (AL) for node classification typically focuses on selecting the most informative nodes for annotation within one or a few large graphs (e.

By Pascal Plettenberg, Denis Huseljic, Andr\'e Alcalde, Bernhard Sick, Josephine M. Thomas

benchmarks

arXiv Machine Learning

Jun 4

Efficient Reasoning on the Edge

arXiv:2603. 16867v2 Announce Type: replace Abstract: Large language models (LLMs) with chain-of-thought reasoning achieve state-of-the-art performance across complex problem-solving tasks, but their verbose reasoning traces and large context requirements make them impractical for edge deployment.

By Yelysei Bondarenko, Thomas Hehn, Rob Hesselink, Romain Lepert, Fabio Valerio Massoli, Evgeny Mironov, Leyla Mirvakhabova, Tribhuvanesh Orekondy, Spyridon Stasis, Andrey Kuzmin, Anna Kuzina, Markus Nagel, Ankita Nayak, Corrado Rainone, Ork de Rooij, Paul N Whatmough, Arash Behboodi, Babak Ehteshami Bejnordi

llms reinforcement-learning fine-tuning efficiency benchmarks

arXiv Machine Learning

Jun 4

Variance-Gated Ensembles: An Epistemic-Aware Framework for Uncertainty Estimation

arXiv:2602. 08142v2 Announce Type: replace Abstract: Machine learning applications require fast and reliable per-sample uncertainty estimation.

By H. Martin Gillis, Isaac Xu, Thomas Trappenberg

benchmarks

arXiv Machine Learning

Jun 4

VentAgent: When LLMs Learn to Breathe -- Multi-Objective Arbitration for ARDS Ventilation

arXiv:2606. 04632v1 Announce Type: new Abstract: Mechanical ventilation for Acute Respiratory Distress Syndrome (ARDS) requires balancing competing physiological goals, including oxygenation, lung protection, and acid-base homeostasis.

By Teqi Hao, Yuxuan Fu, Xiaoyu Tan, Shaojie Shi, Bohao Lv, Yinghui Xu, Xihe Qiu

llms agents reinforcement-learning benchmarks safety

arXiv Machine Learning

Jun 4

Episodic Memory Temporal Consistency for Cooperative Multi-Agent Reinforcement Learning

arXiv:2606. 04492v1 Announce Type: new Abstract: Cooperative Multi-Agent Reinforcement Learning (MARL) frequently suffers from severe reward sparsity and exploration bottlenecks.

By Zicheng Zhao, Yu Lan, Chengzhengxu Li, Zhaohan Zhang, Xiaoming Liu

agents reinforcement-learning efficiency benchmarks

Get the daily briefing