Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,127 stories · RSS feed

arXiv Machine Learning
Oct 19

JAXtari: High-Throughput and Easy-to-Modify Arcade Learning Environment

JAXtari is an open‑source, high‑performance reimplementation of Atari 2600 environments that enables massive GPU parallelization. It allows PPO agents to complete 100 M environment steps on a single A100‑GPU in under an hour, far faster than the CPU‑based ALE. Additionally, JAXtari provides object‑centric observations and a flexible modification system, facilitating research into causal modeling, neuro‑symbolic methods, and rapid agent adaptation.

By Quentin Delfosse, Raban Emunds, Paul Seitz, Sebastian Wette, Jannis Bl\"uml, Daniel Kirn, Dominik Mandok, Kristian Kersting
arXiv AI
Oct 19

Frozen Models, Evolving Expertise: Model-Agnostic Learning from Deployment Experience for Multimodal Medical AI

The paper introduces a model‑agnostic framework that lets frozen large language models (LLMs) and vision‑language models (VLMs) learn from deployment experience without retraining. It does so by adding three external expertise components—a Skill for guiding reasoning and tool use, a Knowledge Memory for storing reliable facts, and a Multimodal Knowledge Base for visual examples—and by using a validation strategy that updates only when new cases improve performance without harming past performance. Experiments on six benchmarks with both open‑weight and closed‑source models show up to a 34.2% performance gain on medical tasks, generalization to unseen cases, and transferability to other models and non‑medical domains.

By Yexiao He, Yucheng Tang, Pengfei Guo, Yufan He, Andriy Myronenko, Can Zhao, Ang Li, Daguang Xu, Dong Yang
arXiv AI
Oct 19

Paint Outside the Box: Synthesizing and Selecting Training Data for Visual Grounding

The paper introduces POBF (Paint Outside the Box and Filter), a framework designed to improve visual grounding in data-scarce settings. POBF synthesizes images by inpainting outside the bounding box to correct label misalignment and employs a filtering scheme that balances hardness and overfitting scores to select the most effective training data. Experiments on four benchmark datasets show that POBF consistently boosts performance, achieving an average 5.83% gain over real-data-only baselines and surpassing leading methods by 2.29%–3.85% in accuracy, while also demonstrating robustness across different generative models, data sizes, and architectures.

By Zilin Du, Haoxin Li, Jianfei Yu, Boyang Li
arXiv AI
Oct 19

Missing value imputation with adversarial random forests - MissARF

The paper introduces MissARF, a new imputation technique that uses adversarial random forests (ARF) for density estimation and data synthesis. MissARF can perform both single and multiple imputations by sampling from the conditional distribution estimated by ARF, conditioned on observed values. Experiments show that MissARF achieves imputation quality comparable to leading methods while offering faster runtimes and no extra cost for multiple imputations.

By Pegah Golchian, Jan Kapar, David S. Watson, Marvin N. Wright
arXiv AI
Oct 19

A Theoretical Analysis of Test-Driven Code Generation

The paper presents a probabilistic framework for two main test‑driven code generation paradigms: selecting code after generation using an execution environment, and generating code conditioned on environment feedback. It formalizes selection heuristics as environment‑aware estimators of code correctness, proving that fuzzy functional similarity estimators provide a stronger inductive bias and higher signal‑to‑noise ratio than functional equivalence estimators. Additionally, it models backprompting as an in‑context approximation of Thompson sampling, deriving a regret bound that explains its limitations due to ambiguous task descriptions, and validates these theoretical insights with experiments on seven open‑weight models across multiple benchmarks, including a new quantum programming benchmark.

By Nicolas Menet, Michael Hersche, Andreas Krause, Abbas Rahimi
arXiv AI
Oct 19

Algorithmic Primitives of Theory of Mind Benchmarks

The paper questions whether Theory of Mind (ToM) benchmarks truly assess theory of mind or merely test parsing and algorithmic state tracking. It introduces PDDL-Mind, a neuro‑symbolic method that parses narratives into the Planning Domain Definition Language (PDDL) and executes candidate actions using predefined update algorithms, yielding program‑verified state traces. Experiments on MMToM, MuMA, and FanToM show that delegating parsing to programs and state tracking to PDDL significantly improves performance, suggesting that current benchmarks mainly penalize unreliable implicit state tracking rather than genuine belief or counterfactual reasoning.

By Qiutong Tony Yi, Wang Bill Zhu, Robin Jia, Jesse Thomason
arXiv AI
Oct 19

MOSAIC: Module Discovery via Sparse Additive Identifiable Causal Learning for Scientific Time Series

MOSAIC is a sparse temporal variational autoencoder that merges causal representation learning with support recovery to identify interpretable latent modules in scientific time series. It leverages regime‑conditioned temporal variation to discover latent variables and uses an additive decoder to recover a sparse set of observations linked to each latent, providing module‑level interpretability. The method offers theoretical guarantees for ANOVA main‑effect identifiability under smooth mixing functions and demonstrates empirical success across diverse domains such as RNA dynamics, solar wind, ENSO climate, industrial process control, and synthetic tokamak data.

By Shicheng Fan, Nour Elhendawy, Jianle Sun, Ke Fang, Kun Zhang, Yihang Wang, Lu Cheng