Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

19,767 stories · RSS feed

arXiv Machine Learning

Jun 25

Learning with a Single Rollout via Monte Carlo Pass@k Critic

arXiv:2606. 25451v1 Announce Type: new Abstract: Estimating token-level advantages in reinforcement learning (RL) for language models remains challenging because scaling up episodic experience collection is expensive.

By Fengdi Che, Yang Liu, Lei Yu, Meng Cao, Tong Che, Rupam Mahmood, Dale Schuurmans

llms reinforcement-learning benchmarks

arXiv Machine Learning

Jun 25

InvestPhilBench: A Multi-Layer Dynamic Benchmark for Evaluating Large Language Model Procedural Reasoning in Expert Investment Philosophy

arXiv:2606. 25984v1 Announce Type: cross Abstract: Large language models are increasingly deployed as investment research assistants, yet no benchmark tests whether they can accurately reconstruct and apply the specific procedural decision frameworks of expert investors.

By Mingguang Chen, Bo Qu

llms benchmarks

arXiv Machine Learning

Jun 25

Certified Robust Invariant Polytope Training in Neural Controlled ODEs

arXiv:2408. 01273v3 Announce Type: replace Abstract: We propose a framework for training neural network controllers with certified robust forward invariant polytopes.

By Akash Harapanahalli, Samuel Coogan

rag benchmarks

arXiv Machine Learning

Jun 25

ATMA: Length-Invariant Language Modeling via Polar Attention and Gated-Delta Compression Memory

arXiv:2606. 25156v1 Announce Type: new Abstract: Modern large language models based on softmax scaled-dot-product attention are constrained by their training sequence length: as the key-value sequence grows, softmax probability mass can dilute across a wider distribution, inducing activation shift and long-context performance collapse.

By Habibullah Akbar

llms benchmarks

arXiv Machine Learning

Jun 25

SDE-Driven Spatio-Temporal Hypergraph Neural Networks for Irregular Longitudinal fMRI Connectome Modeling in Alzheimer's Disease

arXiv:2603. 20452v2 Announce Type: replace Abstract: Longitudinal neuroimaging is essential for modeling disease progression in Alzheimer's disease (AD), yet irregular sampling and missing visits pose substantial challenges for learning reliable temporal representations.

By Ruiying Chen, Yutong Wang, Houliang Zhou, Wei Liang, Yong Chen, Lifang He

efficiency benchmarks

arXiv Machine Learning

Jun 25

The Geometry of Sequential Learning: Lie-Bracket Prediction of Transfer Order

arXiv:2606. 24993v1 Announce Type: new Abstract: Sequential learning is order-dependent: from Pile-style next-token domain adaptation to instruction-SFT and DPO, N candidate sources induce N!

By John Sweeney

llms diffusion reinforcement-learning fine-tuning benchmarks

arXiv Machine Learning

Jun 25

How Modular Is a Frontier Mixture-of-Experts? A Pre-registered Causal Test in Which Apparent Expert Modularity Mostly Dissolves

arXiv:2606. 25092v1 Announce Type: new Abstract: Sparse Mixture-of-Experts (MoE) models route each token to a few of many experts, inviting the hypothesis that experts form functional modules tied to capabilities or languages.

By Tony Salomone, Deep Gandhi, Ali Asaria

efficiency benchmarks

arXiv Machine Learning

Jun 25

From Forecasting Leaderboards to Deployment Decisions: A Fail-Closed Certification Protocol

arXiv:2606. 24996v1 Announce Type: new Abstract: Forecasting leaderboards rank models by predictive quality, but their winners are often read as deployment-ready top-1 advice.

By Geumyoung Kim

benchmarks

arXiv Machine Learning

Jun 25

PERTINENCE: Input-based Opportunistic Neural Network Dynamic Execution

arXiv:2507. 01695v3 Announce Type: replace Abstract: Deep neural networks (DNNs) are widely used for their ability to model complex patterns across domains such as computer vision, speech recognition, and robotics.

By Omkar Shende, Gayathri Ananthanarayanan, Marcello Traiola

computer-vision robotics multimodal benchmarks

arXiv Machine Learning

Jun 25

Reclaim Evaluation: A Lossy Memory Is Worse Than an Empty One

arXiv:2606. 25449v1 Announce Type: cross Abstract: A language model's memory can be worse than having no memory at all.

By Alex Kwon

llms benchmarks

arXiv Machine Learning

Jun 25

Fox in the Henhouse: Supply-Chain Backdoor Attacks Against Reinforcement Learning

arXiv:2505. 19532v2 Announce Type: replace Abstract: The current state-of-the-art backdoor attacks against Reinforcement Learning (RL) rely upon unrealistically permissive access models, that assume the attacker can read (or even write) the victim's policy parameters, observations, or rewards.

By Shijie Liu, Andrew C. Cullen, Paul Montague, Sarah Erfani, Benjamin I. P. Rubinstein

agents reinforcement-learning benchmarks

arXiv Machine Learning

Jun 25

LLM Evolution as an Industry-Scale Ecosystem: A Lifecycle Perspective on Continual Learning

arXiv:2606. 24901v1 Announce Type: new Abstract: Continual learning capability is critical for Industrial LLMs, as deployed models must be continuously updated to meet evolving requirements and environments, rather than repeatedly retrained from scratch.

By Hao Jiang, Enneng Yang, Guojie Zhu, Yibin Chen, Yunkun Xu, Zifu Kou, Jiayi Li, Chong Chen, Zhao Cao, Li Shen

llms reinforcement-learning benchmarks

arXiv Machine Learning

Jun 25

Expresso-AI: Explainable Video-Based Deep Learning Models for Depression Diagnosis

arXiv:2606. 25606v1 Announce Type: cross Abstract: Given the widespread prevalence of depression and its consequential impact on individuals and society, it is crucial to obtain objective measures for early diagnosis and intervention.

By Felipe Moreno, Sharifa Alghowinem, Hae Won Park, Cynthia Breazeal

fine-tuning benchmarks safety

arXiv Machine Learning

Jun 25

Supervised Reinforcement Learning for the Coordination of Distributed Energy Resources

arXiv:2606. 24947v1 Announce Type: new Abstract: The increasing integration of distributed energy resources (DERs) is crucial for power system decarbonization, yet unlocking DERs' flexibility is challenged by their inherent uncertainties and modelling complexity.

By Haoyuan Deng, Yihong Zhou, Thomas Morstyn, Yi Wang

llms reinforcement-learning fine-tuning benchmarks

arXiv Machine Learning

Jun 25

Hierarchical Graph Learning for Calendar Spread Strategies in Commodity Futures Markets

arXiv:2606. 25811v1 Announce Type: cross Abstract: Commodity futures can be represented hierarchically, with underlying assets at the upper level and individual futures contracts at the lower level.

By Yoonsik Hong, Diego Klabjan

benchmarks

arXiv Machine Learning

Jun 25

Model-agnostic Mitigation Strategies of Data Imbalance for Regression

arXiv:2506. 01486v2 Announce Type: replace Abstract: Data imbalance persists as a pervasive challenge in regression tasks, introducing bias in model performance and undermining predictive reliability.

By Jelke Wibbeke, Sebastian Rohjans, Andreas Rauh

benchmarks safety

arXiv Machine Learning

Jun 25

Causal-rCM: A Unified Teacher-Forcing and Self-Forcing Open Recipe for Autoregressive Diffusion Distillation in Streaming Video Generation and Interactive World Models

arXiv:2606. 25473v1 Announce Type: cross Abstract: Autoregressive video diffusion with causal diffusion transformers has emerged as a major paradigm for real-time streaming video generation and action-conditioned interactive world models.

By Kaiwen Zheng, Guande He, Min Zhao, Jintao Zhang, Huayu Chen, Jianfei Chen, Chen-Hsuan Lin, Ming-Yu Liu, Jun Zhu, Qianli Ma

llms diffusion efficiency benchmarks

arXiv Machine Learning

Jun 25

FDN: Interpretable Spatiotemporal Forecasting with Future Decomposition Networks

arXiv:2606. 25201v1 Announce Type: new Abstract: Spatiotemporal systems comprise a collection of spatially distributed yet interdependent entities each generating unique dynamic signals.

By Nicholas Majeske, Ariful Azad

benchmarks safety

arXiv Machine Learning

Jun 25

Do Thinking Tokens Help with Safety?

arXiv:2606. 25013v1 Announce Type: new Abstract: Today's reasoning models use thinking tokens to attain stronger performance on benchmarks than their instruction-tuned counterparts.

By Narutatsu Ri, Abhishek Panigrahi, Sanjeev Arora

llms benchmarks safety

arXiv Machine Learning

Jun 25

RotRNN: Modelling Long Sequences with Rotations

arXiv:2407. 07239v3 Announce Type: replace Abstract: Linear recurrent neural networks, such as State Space Models (SSMs) and Linear Recurrent Units (LRUs), have recently shown state-of-the-art performance on long sequence modelling benchmarks.

By Kai Biegun, Rares Dolga, Jake Cunningham, David Barber

benchmarks

Get the daily briefing