Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,147 stories · RSS feed

arXiv Machine Learning
Oct 9

CoPoE: Multimodal Fusion via Decomposable Disease-Coordinate Product-of-Experts for Missing-Modality Alzheimer's Diagnosis

CoPoE is a new multimodal fusion framework for Alzheimer’s disease diagnosis that maps diverse clinical, imaging, genomic, and biomarker data into a structured latent space divided into four biological and clinical axes: genetic Risk, molecular Pathology, Neurodegeneration, and clinical Stage. Each available modality contributes a diagonal Gaussian expert, and a masked Product-of-Experts architecture fuses only the observed modalities, avoiding synthetic imputation. In extensive experiments on the ADNI dataset, CoPoE outperforms standard missing-modality fusion baselines, achieving higher AUROC, better calibration metrics, and retaining pathology signals even when certain modalities are missing.

By Chihun An, Ikbeom Jang
arXiv Machine Learning
Oct 9

Conditional Transfer from Controlled Pretraining Mixtures to Code

The paper investigates how synthetic tasks influence language‑model pretraining by distinguishing diagnostic, teachable, and transferable signals. In a controlled setting where 70% of the data is general Python and 30% is a mixture of curated synthetic tasks, literature‑derived probes, and OpenCodeInstruct, 14 of 27 tasks show teachability, with curated tasks outperforming literature probes. Curated synthetic data provides conditional value: it boosts downstream performance when mixed with OpenCodeInstruct but is less effective when used alone, and a loss‑based scheduler can misalign short‑term loss reduction with long‑term transfer.

By Ohad Rubin
arXiv Machine Learning
Oct 9

Interval-valued SHAP in Tree-Based Models

The paper introduces an imprecise Dirichlet model (IDM) approach to evaluate the robustness of Shapley values in tree-based models such as decision trees and random forests. By quantifying interval-valued Shapley values—using pessimistic and averaging principles—when a few unannotated instances are added to tree leaves, the authors derive efficient computational methods and extend the framework to Banzhaf values. Experiments and case studies demonstrate how these interval-valued explanations can be applied to debias uninformative features.

By Chenrui Zhu, Vu-Linh Nguyen, Marie-H\'el\`ene Masson, S\'ebastien Destercke
arXiv Machine Learning
Oct 9

Test-Time Compute for Tabular Foundation Models: Mechanisms, Gains, and Limits

The paper investigates how test‑time compute techniques—adaptation, aggregation, and context construction—can improve predictions of pretrained tabular foundation models (TFMs). It introduces DiagScale, a lightweight adaptation method that trains a minuscule fraction of parameters yet matches full fine‑tuning gains, and shows that selective aggregation of predictions from diverse preprocessing variants can reduce error, while broader context expansion offers inconsistent benefits. The study concludes that adaptation and selective aggregation consistently yield benchmark‑level improvements, but their combined use demands more computation, guiding strategy selection based on available resources.

By Kanghui Ning, Marin Bilo\v{s}, James T. Wilson, Yilang Zhang, Kashif Rasul, Dongjin Song, Anderson Schneider, Yuriy Nevmyvaka
arXiv Machine Learning
Oct 9

Large-Scale Benchmarking of Quantum Neural Network Configurations for Financial Time Series Forecasting

The paper reports a large‑scale systematic comparison of 1,368 quantum neural network (QNN) configurations for forecasting the GBP/USD spot exchange rate. By varying encoding methods, ansatz designs, qubit counts, layer depths, and cost functions, the study finds that gate selection and arrangement are more critical than parameter count, and that entanglement is a system‑level property. The top QNN achieves an R² of 0.985, surpassing a classical BiLSTM, while tests on the IQM Emerald device show gate errors and decoherence as major obstacles, with gate choice and circuit depth determining noise resilience.

By Jack Waller, Xing Liang, Dimitrios Makris, Rajagopal Nilavalan
arXiv Machine Learning
Oct 9

Deep Learning vs. Statistical Models for Multi-Horizon Price Forecasting of Second-Hand Electronics: A Systematic Benchmark

The paper introduces the first multi‑horizon benchmark for forecasting resale prices of second‑hand electronics, comparing eleven statistical and deep learning models on a large Polish marketplace dataset covering 100+ smartphone and laptop models from January 2022 to March 2025. Across horizons from 1 to 365 days, N‑BEATS consistently outperforms all statistical baselines, achieving an 8.51% MAPE at 365 days versus 14.94% for the best baseline—a 43% reduction—while all models converge to about 0.72% MAPE for short horizons. A single N‑BEATS model trained for the 365‑day horizon generalizes to all shorter horizons, eliminating the need for horizon‑specific models, and N‑BEATS and N‑HiTS show superior hyperparameter stability.

By Mateusz Buczy\'nski, Micha{\l} Wo\'zniak, Konrad Kaczy\'nski, Anna Wr\'oblewska, Sebastian Kuk
arXiv Machine Learning
Oct 9

Diagnosing and Recovering from Observation-Space Shift at Long-Horizon Skill Seams

The paper investigates Observation‑Space Shift (OSS), a failure mode in long‑horizon robotic manipulation where downstream skills start from states altered by preceding skills. By diagnosing that displaced scene elements—not robot joints or target objects—cause most OSS, the authors develop a detect‑restore‑resume system that monitors task progress, restores the scene, and fine‑tunes the skill to resume. On the BOSS‑44 benchmark, this approach raises full‑chain success from 7.6 % to 26.5 %, outperforming several baselines and demonstrating the value of scene restoration before policy resumption.

By Pranav Wagh, Yu Fang, Yue Yang, Mingyu Ding
arXiv Machine Learning
Oct 9

GHARP: Real-time Gaussian Head Animation from Large-scale Reconstruction Prior

GHARP is a two‑stage method that animates 3D human heads in real time using a large‑scale reconstruction prior. The first offline identity stage builds a compact Gaussian representation of a subject’s geometry and appearance, while the lightweight runtime animation stage predicts expression‑dependent residuals in a semantically structured latent space. A body alignment network further refines the animation by aligning body pose and clothing, enabling state‑of‑the‑art quality on the Ava‑256 benchmark with significantly faster performance and fewer Gaussians.

By Ali Benlalah, Sepehr Johari, Patricia Vitoria, Armin Kappeler, Artem Sevastopolsky, Alexander Jung, Gabriele Fanelli, Kevin Mader, Manuel Breitenstein, Claudia Pl\"uss, Jan R\"uegg, Simon Biland, Thomas Etterlin, Dmitry Kostiaev, Mathias Deschler, Brian Amberg, Sebastian Martin
arXiv Machine Learning
Oct 9

Lapras: Latent Reasoning for Time Series Language Models

Lapras is a post‑training framework for Time Series Language Models that replaces explicit Chain‑of‑Thought reasoning with continuous latent thoughts in a joint time‑series‑language space. By self‑distilling from a teacher that generates textual rationales, the student model aligns its hidden states at the answer stage, transferring reasoning ability while producing only the final answer in text. Experiments on four TSLM backbones and five benchmarks show Lapras boosts average F1 by up to 10.79% over explicit CoT and reduces token generation by 23.9×, while still enabling readable reasoning traces through standard decoding.

By Yuliang Chen, Yu Yvonne Wu, Patrick Langer, Arvind Pillai, Sudarshan Regmi, Martin Maritsch, Juncheng Liu, Robert Jakob, Thomas Kaar, Tess Z. Griffin, Lisa Marsch, Michael V. Heinz, Nicholas C. Jacobson, Andrew Campbell
arXiv Machine Learning
Oct 9

BioBigBird: A Sparse Attention Model for Long-Range Dependency Processing in Biomedical Text

BioBigBird is a bidirectional language model tailored for biomedical text that uses sparse attention to process sequences up to 4096 tokens, addressing the limited context windows of existing domain-specific LLMs. It is pre‑trained on extensive biomedical literature and clinical data, with a multi‑stage training process to reduce noise and a multi‑task learning framework that jointly optimizes Named Entity Recognition and Relation Extraction. Evaluations on the BLURB benchmark show that the MTL‑enhanced BioBigBird achieves highly competitive results against state‑of‑the‑art models, and the models are publicly available on Hugging Face.

By Roshan Balaji, Pavan Kumar S, Vasudev Gupta, Sreejith N, Keerthana Sridhar, Nirav Bhatt
arXiv Machine Learning
Oct 9

Open-Vocabulary Audio-Visual Event Localization via Complex-Valued Fusion

The paper introduces a novel four‑stream complex‑valued architecture for Open‑Vocabulary Audio‑Visual Event Localization (OV‑AVEL). By embedding visual, audio, and textual modalities into a shared space and computing complex‑valued similarities—real parts from standard embeddings and imaginary parts from companion streams (iHSV for vision and CycleGAN‑translated phase for audio)—the method learns fusion via a complex‑valued neural network while keeping encoders frozen. This approach achieves new state‑of‑the‑art results on OV‑AVEBench and a modified AVE dataset, outperforming previous fine‑tuned baselines by significant margins. "whyItMatters":"The proposed complex‑valued fusion strategy demonstrates that incorporating phase information and learning fusion in the complex domain can substantially improve open‑vocabulary event localization performance, setting new benchmarks for the task."

By Anirudh Praveen, Koteswar Rao Jerripothula, Pratik Joshi, Aveen Dayal, Neela Sawant
arXiv Machine Learning
Oct 9

Beyond Spatio-Temporal Priors: A Generalizable Approach for Dense Correspondence Matching

The paper introduces FreeMatching, a framework that moves beyond traditional spatio-temporal priors for dense correspondence matching. It combines generative and semantic representations with diverse supervision sources, and uses teacher‑guided refinement to improve identity‑preserving correspondence in image editing and reference‑guided generation tasks. Experiments show that a single FreeMatching model outperforms existing methods on challenging image pairs while maintaining strong performance on classical benchmarks, and its scores correlate with human judgment of identity preservation.

By Luping Liu, Bingyi Kang, Yifan Wang, Dong Xu
arXiv Machine Learning
Oct 9

LIME: Link-based User-item Interaction Modeling with Decoupled XOR Attention for Efficient Test Time Scaling

LIME is a new recommendation system architecture that tackles the high computational cost of transformers by introducing low‑rank link embeddings and a linear attention mechanism called LIME‑XOR. These innovations decouple user and candidate interactions, allowing pre‑computation of attention weights and reducing inference complexity to be nearly independent of candidate set size and linear in user sequence length. Experiments on public and industrial datasets demonstrate that LIME matches transformer performance while achieving a tenfold speedup on large candidate sets or long sequences, and it has been shown to improve user engagement on a major recommendation platform.

By Yang Liu, Yunjiang Jiang, Ayush Agarwal, Yihan Wu, Haoran Liu, Bi Xue
arXiv Machine Learning
Oct 9

V-ECE: Estimating General Expected Calibration Errors

The paper introduces V-ECE, a variational estimator for general Expected Calibration Errors (ECE) that avoids binning and works with any convex divergence, including Lp distances. By using prediction‑dependent proper scores and a residual‑fitted temperature scaling via gradient boosting, V‑ECE provides a lower‑bound estimate of true calibration error. Experiments on semi‑synthetic benchmarks show V‑ECE outperforms existing binary and multiclass estimators, with additional theoretical insights on bias and confidence estimation.

By Eug\`ene Berta, Sacha Braun, Francis Bach, Michael I. Jordan, David Holzm\"uller
arXiv Computation and Language
Oct 9

Diffu-LoRA: A Novel Low-Rank Adaptation for Personalized Diffusion Models

Diffu-LoRA is a parameter‑efficient method for personalizing text‑to‑image diffusion models. It inserts trainable low‑rank components into Transformer linear layers and assigns a learnable gate to each component, allowing the model to learn how to distribute adaptation capacity across layers. Using bilevel optimization and progressive pruning, Diffu-LoRA keeps the pretrained backbone frozen while improving subject fidelity and prompt alignment compared to fine‑tuning baselines.

By Tianjing Li, Wei Zhu
arXiv Computation and Language
Oct 9

Lossy Compressive Text Autoencoders

The paper introduces a lossy compressive text autoencoder that learns a compressed latent representation of text by applying residual downscaling and upscaling along the time axis, culminating in a low‑dimensional discrete bottleneck. The authors evaluate various quantization methods, training objectives, and datasets, measuring reconstruction quality with BLEU and LLM‑based semantic judgments, and also test downstream performance on question‑answering and semantic similarity tasks. Their results show that the compressed representations achieve compression rates comparable to lossless algorithms (2.24 bits per byte on web text) while maintaining strong reconstruction and downstream task performance.

By Vinko Sabol\v{c}ec, Angelos Katharopoulos, David Grangier
arXiv Computation and Language
Oct 9

Clinician use of language models diverges from how the models are evaluated

Large language model assistants are being deployed to clinicians, but their readiness is judged mainly by benchmark scores that are based on examination questions or curated cases. This study analyzed 127,833 queries from 6,342 healthcare professionals across 35 specialties, finding that most requests were for documentation, administration, and knowledge retrieval, with a significant portion of queries unanswerable. When compared to 58 public benchmarks, the median benchmark lacked documentation requests and only shared 31% of the real task mix, indicating that current benchmarks poorly reflect actual clinical use.

By Krithik Vishwanath, Haitong Lin, Anton Alyakin, Jin Vivian Lee, D. Brock Hewitt, Jie J. Yao, William Robert Small, Hammad A. Khan, Cordelia Orillac, Aakaash Varma, Brandon Ye, Daniel Alexander Alber, Gustavo Stolovitzky, Batia Wiesenfeld, Oded Nov, Wei Wu, Kang Zhang, Yindalon Aphinyanaphongs, Tim Requarth, Eric Karl Oermann, The International Digital Twin Consortium in Healthcare, Medicine
arXiv Computation and Language
Oct 9

SignRAG: Unified Retrieval-Augmented Gloss-Free Sign Language Translation

SignRAG is a unified framework for gloss‑free sign language translation that combines hierarchical pretraining, target‑domain retrieval augmentation, and retrieval‑aware reinforcement fine‑tuning. It first learns linguistically grounded sign representations, aligns a sign encoder with a decoder‑only large language model, and then uses a retrieval gallery to provide instance‑specific translation cues. Retrieval Utility‑Guided Reinforcement Fine‑Tuning (RUG‑RFT) balances translation quality and retrieval usefulness, leading to state‑of‑the‑art results, including the first gloss‑free approach to outperform gloss‑supervised methods on CSL‑Daily.

By Zhi Rao, Yucheng Zhou, Qianran Sun, Yiqing Huang, Longcan Yuan, Jiayi Hou, Chengwen Yao, Lin Cheng, Donghui Sun, Xiaoxin Chen, Jun Wan