Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,322 stories · RSS feed

Hugging Face Trending Papers
Sep 21

HappyWorld-Bench

Evaluating world models requires assessing both the quality of the worlds they generate and their consistency and responsiveness under exploration, interaction, and modification. We introduce HappyWor...

arXiv Computer Vision
Sep 21

Uncertainty-driven training for three-dimensional calibrated lung nodule classification

The paper introduces an uncertainty‑driven training framework for 3D CT lung nodule classification that uses validation‑based uncertainty estimates to reweight the loss, aiming to improve predictive performance and probability calibration. Two uncertainty quantification methods—Monte Carlo Dropout and Evidential Deep Learning—are evaluated across multiple backbone architectures (ResNet, DenseNet, EfficientNet, ViT, Swin) on the LIDC‑IDRI and NoduleMNIST3D datasets. The approach yields comparable classification accuracy to conventional training while substantially reducing expected calibration error, especially on convolutional backbones, and shows that simple temperature scaling can also achieve strong calibration.

By Giuseppe Tripodi, Alessandro De Rosis, Saleh Rezaeiravesh
arXiv AI
Sep 21

CaLR: Causal Latent Revision for Robust Diffusion Reasoning

CaLR: Causal Latent Revision for Robust Diffusion Reasoning proposes a framework that combines autoregressive and diffusion language models by reformulating reasoning as constrained latent optimization. The method uses a causal topology matrix from an expert model and implicit differentiation to perform gradient-guided thought revision, enforcing logical consistency and allowing dynamic self-correction of intermediate steps during parallel generation. Empirical results show CaLR achieves state‑of‑the‑art performance on complex benchmarks, surpassing strong autoregressive baselines and demonstrating superior robustness in constrained tasks such as Sudoku.

By Wei Cai, Jian Zhao, Yuchen Yuan, Xuelong Li
arXiv AI
Sep 21

Geometry of Values: Task Vector Composition for Ethical Preference Alignment in Language Models

The paper introduces a 12,000-instance dataset of two-option moral dilemmas covering three pairwise value conflicts—Honesty vs. Justice, Justice vs. Autonomy, and Autonomy vs. Honesty—translated into Hindi, Arabic, Spanish, and Chinese to test cross‑lingual behavior. Benchmarking on GPT‑5‑mini shows a consistent preference for Honesty over Autonomy across all languages when no policy is provided, while Llama‑3.2‑1/3B models exhibit a strong first‑option bias that is largely eliminated by plain fine‑tuning or Direct Preference Optimization, raising accuracy above 98%. The authors propose a task vector transfer method that orthogonalizes value preference vectors with respect to general instruction‑following vectors, effectively isolating specific value directions and enabling task arithmetic to flip a model’s stance.

By Utkarsh Agarwal, Monojit Choudhury
arXiv AI
Sep 21

Learning Cardiac Features: ECG Biometrics Across Time and~Exercise

The study investigates ECG biometrics by training a Siamese ResNet with late multi-lead fusion on a large dataset from cardiopulmonary exercise tests. It evaluates the model under realistic conditions, including exercise-induced stress and cross-session variability, achieving an intra-session rest-to-peak EER of 1.7% and a state‑of‑the‑art 3.9% on the CYBHi dataset. The results demonstrate that an intrinsic cardiac signature remains robust to physiological and temporal drift.

By Luca Thiebaud (AMU, AMU SCI, DIAPRO, LIS), Paul Chauchat (AMU SCI, AMU, LIS, DIAPRO), Mustapha Ouladsine (AMU SCI, AMU, LIS, DIAPRO), St\'ephane Delliaux (AMU, APHM, C2VN)
arXiv AI
Sep 21

Neural Cellular Automata Learn General Features in their Hidden Channels

Neural Cellular Automata (NCAs) are shown to learn general, scale‑invariant topological primitives in their hidden channels, which can be transferred from a teacher to a student model for few‑shot learning. The study introduces a transfer‑learning mechanism that injects pretrained hidden states into a student, improving early optimization and outperforming recurrent and feed‑forward baselines on MNIST benchmarks with only ~9,800 parameters. Mechanistic analysis reveals that hidden channels decouple feature extraction from classification, converging to mutually orthogonal states that absorb morphological complexity.

By Etienne Guichard, Stefano Nichele
arXiv Machine Learning
Sep 21

A Lightweight Plug-in Gate for Transformer-Based Time-Series Forecasters

The paper introduces a lightweight pre‑encoder gate for Transformer‑based time‑series forecasters, which assigns sigmoid scores to covariate representations before they enter the encoder. The gate is evaluated as a plug‑in for models such as TimeXer, iTransformer, and PatchTST on datasets including ETTm1, ETTm2, Traffic, Energy, and ILI, showing competitive performance and the ability to regulate covariate admission via a usage penalty. Experiments also explore gate placement, initialization, and feature importance using VIF‑informed permutation diagnostics.

By Hongkai Zhuang, Tao Huang, Chen Hou