Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

19,504 stories · RSS feed

arXiv AI
Jun 29

Uncertainty-Aware Reward Discounting for Mitigating Reward Hacking

arXiv:2604. 26360v2 Announce Type: replace-cross Abstract: Reinforcement learning from human feedback (RLHF) systems face a compounding alignment challenge: not only are learned reward models uncertain about unseen state-action pairs, but the human preference annotations they are trained on are themselves inconsistent, context-dependent, and noisy.

By Disha Singha
arXiv Machine Learning
Jun 29

Mosaic: A Benchmark Suite for Differentiable Physics Solvers

arXiv:2606. 27895v1 Announce Type: cross Abstract: Differentiable partial differential equation (PDE) solvers underpin solver-in-the-loop ML training, gradient-based optimal control, and inverse problems, yet the practical cost of obtaining correct, usable gradients from a given solver on a given problem is largely undocumented.

By Andrin Rehmann, Heiko Zimmermann, Dion H\"afner
arXiv AI
Jun 29

Drop-Then-Recovery: How Redundant Are Vision-Language-Action Models?

arXiv:2606. 27755v1 Announce Type: cross Abstract: Vision-Language-Action (VLA) models enable instruction-driven robotic manipulation, but they inherit oversized language backbones from pretrained VLMs whose capacity far exceeds what is needed for short robotic instructions.

By Guoheng Sun, Kaixi Feng, Shwai He, Xiaochuan Gong, Yexiao He, Ziyao Wang, Zheyu Shen, Wanghao Ye, Ramana Rao Kompella, Gaowen Liu, Ang Li
arXiv AI
Jun 29

CBD: API-Only LLM Black-Box Unlearning through Controlled Behavioral Divergence

arXiv:2606. 27683v1 Announce Type: cross Abstract: Edge devices increasingly invoke large language models (LLMs) through API services for context aware edge intelligence, while edge generated data may be collected to improve LLMs and may introduce sensitive, copyrighted, harmful, or outdated information into model behavior.

By Zhiqiang Xie, Yijing Lin, Zhipeng Gao, Dong In Kim