Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,306 stories · RSS feed

arXiv AI
Sep 30

MGSM-Pro: A Simple Strategy for Robust Multilingual Mathematical Reasoning Evaluation

arXiv:2601.21225v4 Announce Type: replace-cross Abstract: Large language models have made substantial progress in mathematical reasoning. However, benchmark development for multilingual evaluation ha...

By Tianyi Xu, Kosei Uemura, Alfred Malengo Kondoro, Tadesse Destaw Belay, Catherine Nana Nyaah Essuman, Ifeoma Okoh, Ganiyat Afolabi, Ayodele Awokoya, David Ifeoluwa Adelani
arXiv AI
Sep 30

Alignment Forecasting: Predicting Misalignment From Training Data

The paper introduces Alignment Forecasting, a method for predicting whether fine‑tuning a language model on a given dataset will increase specific alignment failures such as deception or sycophancy. It presents ALIGNMENTFORECASTBENCH, a benchmark of over 5,000 forecasting questions across many models, datasets, and failure modes, and shows that a simple forecasting scaffold using an LLM’s assessment of dataset bias can outperform baseline forecasters. The authors demonstrate that filtering out high‑risk training examples identified by the forecaster can improve alignment in multiple‑choice evaluations, though benefits in open‑ended conversations remain uncertain.

By Chen Yueh-Han, Bruce W. Lee, Ilia Sucholutsky, Tomek Korbak
arXiv Machine Learning
Sep 30

Reward Valuation in Large Language Models: Causal Induction of Anhedonia

The study investigates whether large language models (LLMs) exhibit reward valuation mechanisms analogous to human anhedonia by applying clinical tests designed for major depressive disorder. Researchers identified reward‑anticipatory units in state‑of‑the‑art AI models, showed that perturbing these units predicts Nucleus Accumbens activity, and caused the models to choose low‑effort, low‑reward tasks—mirroring human anhedonia. The findings suggest that specific reward‑valuation circuits in AI can functionally resemble those in humans, providing a mechanistic bridge between computational and neurobiological models of motivation.

By Melika Honarmand, Samin Mahdipour Aghabagher, Martin Schrimpf
arXiv Computer Vision
Sep 30

ByteTraX: Enhancing the ByteTrack Architecture with Optimised Thresholding

ByteTraX is a lightweight enhancement to the ByteTrack multi‑object tracking architecture that introduces a single unified matching threshold and stricter track initiation criteria to reduce erroneous track reclassification and identity switches. The method yields consistent performance gains across several benchmarks—GMOT‑40, LC‑MOT, SportsMOT, TeamTrack, DAMUNT, and DeepSea‑MOT—while boosting processing speed by over 10%. Quantitatively, ByteTraX achieves more than a 40% drop in identity switches, with mean improvements of 3.6 in HOTA, 5.6 in IDF1, and 6.3 FPS.

By Thomas A. O'Shea-Wheller
arXiv Computer Vision
Sep 30

SoL-Refiner: Speed-of-Light One-Step Refinement for High-Resolution Video

arXiv:2609.37969v1 Announce Type: new Abstract: High-resolution video generation is expensive, as its cost grows rapidly with the number of spatiotemporal tokens. A practical alternative first genera...

By Haozhe Liu, Tian Ye, Shuchen Xue, Yitong Li, Junsong Chen, Haopeng Li, Jincheng Yu, Duomin Wang, Ruihua Zhang, Lei Zhu, Song Han, Enze Xie
arXiv AI
Sep 30

AnyAct: Universal Action for Self-Evolving Agents

AnyAct introduces a universal action layer that consolidates diverse tool capabilities into a self‑evolving action space for AI agents operating in open‑world environments. It tackles the scale dilemma, tool non‑stationarity, and heterogeneous feedback by using hierarchical progressive retrieval and test‑time reliability evolution, while a heterogeneous observation grounding module unifies multi‑modal feedback. Evaluations on LiveMCPBench and the newly created OSMCP benchmark show state‑of‑the‑art performance, with significant gains in task success rate and reduced execution steps, especially for models with limited native capabilities.

By Lingrui Xu, Yangqin Jiang, Jiachang Zhang, Xubin Ren, Chao Huang
arXiv Machine Learning
Sep 30

THEIA: A Multimodal Dataset and Benchmark for Vision-Language Analysis of Layout

THEIA is a multimodal dataset that pairs thousands of analog circuit layout images with question‑answer conversations, and it introduces a benchmark using a fine‑tuned vision‑language model to analyze GDSII files. The dataset and benchmark enable designers to interact with and query physical layouts as intuitive, meaningful entities. Experiments on five realistic tasks show the fine‑tuned model outperforms general‑purpose vision‑language models by up to 73%, revealing a significant gap between general multimodal reasoning and domain‑specific layout understanding.

By Giuseppe Chiari, Michele Piccoli, Federico Viola, Davide Zoni
arXiv Machine Learning
Sep 30

NeuronSifter: Intervention Planning in CNS Microenvironments

NeuronSifter is a framework for planning interventions in central nervous system microenvironments by converting treatment regimens into state‑conditional target‑occupancy fields and propagating them through microenvironment dynamics. It selects measurements based on their expected reduction in intervention loss, integrating typed outcomes into a unified posterior. In synthetic Alzheimer’s disease scenarios, occupancy conditioning improves trajectory probability scores and intervention ordering accuracy, and decision‑directed acquisition reduces terminal risk compared to a Bayesian experimental design planner.

By Haowei Xu, Wanyi Fu, Hongbin Han, Zhaoheng Xie