Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

24,477 stories · RSS feed

arXiv AI

Jun 3

Building Reliable Long-Form Generation via Hallucination Rejection Sampling

arXiv:2606. 03628v1 Announce Type: cross Abstract: Large language models (LLMs) have achieved remarkable progress in open-ended text generation, yet they remain prone to hallucinating incorrect or unsupported content, which undermines their reliability.

By Lin Li, Georgia Channing, Suhaas M Bhat, Gabriel Davis Jones, Yarin Gal

llms benchmarks safety

arXiv Machine Learning

Jun 3

KVarN: Variance-Normalized KV-Cache Quantization Mitigates Error Accumulation in Reasoning Tasks

arXiv:2606. 03458v1 Announce Type: new Abstract: Test-time scaling is a powerful approach to obtain better reasoning in large language models, but it becomes memory-bottlenecked during long-horizon decoding, as the KV-cache grows.

By Lorenz K. Muller, Philippe Bich, Chiara Boretti, Hyun-Min Chang, Jiawei Zhuang, Lukas Cavigelli

llms efficiency benchmarks

arXiv AI

Jun 3

Plan, Verify and Fill: A Structured Parallel Decoding Approach for Diffusion Language Models

arXiv:2601. 12247v3 Announce Type: replace-cross Abstract: Diffusion Language Models (DLMs) present a promising non-sequential paradigm for text generation, distinct from standard autoregressive (AR) approaches.

By Miao Li, Hanyang Jiang, Sikai Cheng, Hengyu Fu, Yuhang Cai, Baihe Huang, Tinghan Ye, Xuanzhou Chen, Pascal Van Hentenryck

llms diffusion benchmarks

arXiv AI

Jun 3

VidMsg: A Benchmark for Implicit Message Inference in Short Videos

arXiv:2606. 03635v1 Announce Type: cross Abstract: Understanding short online videos involves more than identifying visible objects and actions; video makers often include an underlying message or purpose in the clip.

By Issar Tzachor, Michael Green, Rami Ben-Ari

llms benchmarks safety

arXiv AI

Jun 3

Vision-OPD: Learning to See Fine Details for Multimodal LLMs via On-Policy Self-Distillation

arXiv:2605. 18740v4 Announce Type: replace-cross Abstract: Multimodal Large Language Models (MLLMs) still struggle with fine-grained visual understanding, where answers often depend on small but decisive evidence in the full image.

By Qianhao Yuan, Jie Lou, Xing Yu, Hongyu Lin, Le Sun, Xianpei Han, Yaojie Lu

llms agents efficiency multimodal benchmarks

arXiv AI

Jun 3

Scalable On-Hardware Training of Quantum Neural Networks and Application to Clinical Data Imputation

arXiv:2606. 03517v1 Announce Type: cross Abstract: Training quantum neural networks (QNNs) on quantum hardware is currently bottlenecked by the cost of gradient estimation: standard parameter-shift methods require a number of circuit evaluations that grows quadratically with the number of trainable parameters, making hardware-based optimisation impractical beyond small system sizes.

By Natansh Mathur, Panagiotis Kl. Barkoutsos, Masako Yamada, Martin Roetteler, Iordanis Kerenidis

benchmarks

arXiv Machine Learning

Jun 3

Right Makes Might: Aligning Verified Hidden States Empowers RL Reasoning

arXiv:2606. 03234v1 Announce Type: new Abstract: Reinforcement Learning from Verifiable Rewards (RLVR) has become the dominant approach for improving mathematical reasoning in large language models, yet current methods reduce each correct rollout to a single reward bit, ignoring the geometric structure shared among their hidden states.

By Ziyue Wang, Aomufei Yuan, Yongfu Zhu, Shuai Dong, Wenpu Liu, Yiran Yao, Weichu Xie, Yuqi Xu, Caoyuan Ma, Wenqi Shao, Xiaoying Zhang, Nan Duan, Jiaqi Wang

llms reinforcement-learning benchmarks safety

arXiv AI

Jun 3

Exact Stiefel Optimization for Probabilistic PLS: Closed-Form Updates, Error Bounds, and Calibrated Uncertainty

arXiv:2605. 11607v2 Announce Type: replace-cross Abstract: Probabilistic partial least squares (PPLS) is a central likelihood-based model for two-view learning when one needs both interpretable latent factors and calibrated uncertainty.

By Haoran Hu, Xingce Wang

benchmarks safety

arXiv Machine Learning

Jun 3

FinStressTS: A Parametric Synthetic Benchmark for Time-Series Forecasting in Finance

arXiv:2606. 03184v1 Announce Type: cross Abstract: Financial forecasting is difficult due to low signal-to-noise ratios, latent factors, heavy tails, regime shifts, and jumps.

By Jiaze Sun, Kelvin J. L. Koa, Ruiyang Ni, Yize Liu, Haonan Chen, Ke-Wei Huang

llms multimodal benchmarks safety

arXiv Machine Learning

Jun 3

Learning Temporal Causal Structure via Smooth Differentiable Optimization

arXiv:2606. 03227v1 Announce Type: new Abstract: Causal discovery with instantaneous effects in multivariate time series is challenging, as the instantaneous structure must be acyclic.

By Tong Zhao, Ce Guo, Wayne Luk, Emil Lupu, Ray Dipojjwal

benchmarks

arXiv AI

Jun 3

InftyThink+: Effective and Efficient Infinite-Horizon Reasoning via Reinforcement Learning

arXiv:2602. 06960v3 Announce Type: replace-cross Abstract: Large reasoning models achieve strong performance by scaling inference-time chain-of-thought, but this paradigm suffers from quadratic cost, context length limits, and degraded reasoning due to lost-in-the-middle effects.

By Yuchen Yan, Liang Jiang, Jin Jiang, Shuaicheng Li, Zujie Wen, Zhiqiang Zhang, Jun Zhou, Jian Shao, Yueting Zhuang, Yongliang Shen

nlp reinforcement-learning efficiency benchmarks

arXiv AI

Jun 3

When to Re-Plan: Subgoal Persistence in Hierarchical Latent Reasoning

arXiv:2606. 03741v1 Announce Type: new Abstract: Long-horizon reasoning requires a system to commit to medium-horizon intent without becoming rigid: re-plan too often and computation never coheres into multi-step structure; commit too long and the plan goes stale.

By Ayushi Chadha

benchmarks safety

arXiv AI

Jun 3

Large AI Models in Dental Healthcare: From General-Purpose Systems to Domain-Specific Foundation Models

arXiv:2606. 02914v1 Announce Type: new Abstract: Background: Oral diseases affect nearly 3.

By Sema Helali, Lina Abu Nadab, Sausan Alqawas, Alaa Abd-Alrazaq, Faleh Tamimi, Rafat Damseh

agents computer-vision multimodal benchmarks safety

arXiv AI

Jun 3

Hallucinations as Orthogonal Noise: Inference-Time Manifold Alignment via Dynamic Contextual Orthogonalization

arXiv:2606. 03022v1 Announce Type: cross Abstract: Hallucination in Large Language Models (LLMs), characterized by the generation of content inconsistent with contextual facts or logical constraints -- remains a persistent challenge for reliable deployment.

By Mingkuan Zhao, Wentao Hu, Tianchen Huang, Yuheng Min, Suquan Chen, Yide Gao, Yanbo Zhai, Shuangyong Song, Xuelong Li

llms benchmarks safety

arXiv AI

Jun 3

Testing LLM Arithmetic Reasoning Generalization with Automatic Numeric-Remapping Attacks

arXiv:2606. 03606v1 Announce Type: cross Abstract: Large language models achieve strong performance on arithmetic reasoning benchmarks, and one common response to arithmetic brittleness is to delegate computation to code.

By Malia Barker, Bishal Lakha, Edoardo Serra, Francesco Gullo

llms benchmarks

arXiv Machine Learning

Jun 3

Synthetic Hallucinations, Real Gains: Hard Negatives from Frontier Models for FIM Hallucination Mitigation

arXiv:2606. 03130v1 Announce Type: new Abstract: Small open-source code models that power IDE autocomplete still emit hallucinated Fill-in-the-Middle (FIM) completions: syntactically natural calls to methods, parameters, variables, and imports that do not exist in the surrounding project.

By Mahdi Erfanian, Nelson Daniel Troncoso, Aashna Garg, Amabel Gale, Xiaoyu Liu, Pareesa Ameneh Golnari, Shengyu Fu

llms reinforcement-learning fine-tuning benchmarks safety

arXiv AI

Jun 3

Don't Gamble, GAMBLe: An Analytical Framework for AI-Driven Research Systems

arXiv:2606. 02863v1 Announce Type: new Abstract: AI-Driven Research Systems (ADRS) -- systems coupling LLMs with automated evaluation to discover algorithms, proofs, and designs -- are being optimized and adopted across domains, but the tools to analyze them have not kept pace.

By Marquita Ellis, Paul Castro

llms benchmarks

arXiv AI

Jun 3

QUBRIC: Co-Designing Queries and Rubrics for RL Beyond Verifiable Rewards

arXiv:2606. 03968v1 Announce Type: cross Abstract: Rubric-based RL is a promising route for extending reinforcement learning beyond verifiable rewards, yet existing methods optimize rubrics while treating the query distribution as fixed.

By Rongzhi Zhang, Rui Feng, Zhihan Zhang, Jingfeng Yang, Qingyu Yin, Xin Liu, Zixuan Zhang, Priyanka Nigam, Bing Yin, Tuo Zhao, Chao Zhang

reinforcement-learning fine-tuning benchmarks

arXiv Machine Learning

Jun 3

Fast Organic Crystal Structure Prediction with Unit Cell Flow Matching

arXiv:2606. 03199v1 Announce Type: new Abstract: Organic crystal structure prediction (CSP) is a requirement for computational modelling of organic solids, but traditionally costs several CPU-years per molecule.

By Alston Lo, Luka Mucko, Austin H. Cheng, Andy Cai, Alastair J. A. Price, Wojciech Matusik, Al\'an Aspuru-Guzik

diffusion benchmarks safety

arXiv AI

Jun 3

ROBUST-WT: Robust Uncertainty-aware Segmentation Transform via Whitening and Training Enhancements

arXiv:2606. 03069v1 Announce Type: cross Abstract: Generalized segmentation of medical images prevents performance degradation when different imaging devices and clinical protocols are used across multiple domains.

By Aqsa Naseer, Maryam Bibi, Syeda Samiya Urooj, Muhammad Khurram Shahzad

computer-vision efficiency benchmarks

Get the daily briefing