Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,620 stories · RSS feed

arXiv AI
Sep 10

DUA-D2C: Dynamic Uncertainty Aware Method for Overfitting Remediation in Deep Learning

The paper introduces DUA-D2C, a Dynamic Uncertainty-Aware Divide2Conquer method that improves overfitting remediation in deep learning. It refines the traditional Divide2Conquer approach by dynamically weighting subset models based on a composite score of accuracy and normalized prediction entropy, allowing the central model to learn more from generalizable and confident edge models. The authors provide theoretical justification, show reduced model variance, and demonstrate significant generalization gains across image, audio, and text benchmarks, even when combined with standard regularizers like Dropout.

By Md. Saiful Bari Siddiqui, Md Mohaiminul Islam, Md. Golam Rabiul Alam
arXiv AI
Sep 10

CogniDir: Combating Cognitive Malicious Comments via Adaptive Distributional Learning for Robust Fake News Detection

CogniDir is an adaptive distributional learning framework designed to improve fake news detection against new psychologically grounded malicious comments generated by Large Language Models. It reframes robust detection as a dynamic data mixture optimization problem, using cognitive psychology to formalize adversarial paradigms and an information‑theoretic score to guide adaptive sampling of training data. Experiments on three benchmarks show that CogniDir achieves state‑of‑the‑art robustness, boosting F1 scores by up to 17.9% over existing baselines under heterogeneous AI‑generated attacks.

By Zhao Tong, Chunlin Gong, Yimeng Gu, Haichao Shi, Qiang Liu, Shu Wu, Xingcheng Xu, Xiao-Yu Zhang
arXiv AI
Sep 10

TimeBlind: A Spatio-Temporal Compositionality Benchmark for Video LLMs

TimeBlind is a diagnostic benchmark designed to evaluate fine‑grained spatio‑temporal compositionality in video large language models (LLMs). It categorizes temporal understanding into three levels—atomic event recognition, event property characterization, and reasoning about event interdependencies—and uses a minimal‑pairs paradigm where video pairs share identical static content but differ only in temporal structure. Across 20 state‑of‑the‑art MLLMs tested on 600 curated instances, the best model achieved only 48.2% instance accuracy, far below human performance of 98.2%, highlighting a reliance on static visual shortcuts rather than true temporal reasoning.

By Baiqi Li, Kangyi Zhao, Ce Zhang, Chancharik Mitra, Jean de Dieu Nyandwi, Gedas Bertasius
arXiv AI
Sep 10

Boosting LLM Reasoning via Human-Inspired Reward Shaping

The paper introduces T2T (Thickening-to-Thinning), a dynamic reward framework for large language models that mimics human learning by separating exploration and consolidation phases. During incorrect attempts, T2T encourages exploration to broaden the search space, while after correct solutions it applies length penalties to promote concise reasoning. Experiments on mathematical benchmarks across five mainstream LLMs show that T2T outperforms standard GRPO and recent baselines, improving overall reasoning performance.

By Wenze Lin, Zhen Yang, Xitai Jiang, Xiaoteng Ma, Gao Huang
arXiv AI
Sep 10

Explainable Token-level Noise Filtering for LLM Fine-tuning Datasets

The paper introduces XTF, an explainable token‑level noise filtering framework for fine‑tuning large language models. XTF breaks down token contributions into reasoning importance, knowledge novelty, and task relevance, scores them, and masks gradients of noisy tokens to improve fine‑tuning. Experiments on math, code, and medicine tasks across seven LLMs show up to a 13.7% performance boost over standard fine‑tuning.

By Yuchen Yang, Wenze Lin, Enhao Huang, Zhixuan Chu, Hongbin Zhou, Lan Tao, Yiming Li, Zhan Qin, Kui Ren
arXiv AI
Sep 10

US-JEPA: A Joint Embedding Predictive Architecture for Ultrasound

US-JEPA introduces a self‑supervised framework for ultrasound imaging that predicts masked latent representations instead of raw pixels, using a frozen, domain‑specific teacher to provide stable targets. This approach avoids the hyperparameter sensitivity and computational cost of traditional online teachers, enabling the student model to build upon the teacher’s semantic priors. The authors benchmark US‑JEPA against all publicly available ultrasound foundation models on UltraBench, showing competitive or superior performance across multiple organs and pathological conditions under linear probing.

By Ashwath Radhachandran, Vedrana Ivezi\'c, Shreeram Athreya, Corey W. Arnold, William Speier
arXiv AI
Sep 10

PersonaTeaming: Supporting Persona-Driven Red-Teaming for Generative AI

PersonaTeaming introduces a workflow that incorporates personas into adversarial prompt generation for generative AI, achieving higher attack success rates than the state‑of‑the‑art RainbowPlus while preserving prompt diversity. The system is extended into a user‑facing playground that lets red‑teamers create their own personas and collaborate with AI to refine prompts, fostering diverse strategies. A user study with 11 industry practitioners found the playground produced useful outputs and encouraged out‑of‑the‑box thinking, even when suggestions were not strictly followed.

By Wesley Hanwen Deng, Mingxi Yan, Sunnie S. Y. Kim, Akshita Jha, Lauren Wilcox, Kenneth Holstein, Motahhare Eslami, Leon A. Gatys
arXiv Computation and Language
Sep 10

Qwen-Audio-3.0-ASR Technical Report

The Qwen-Audio-3.0-ASR Technical Report introduces a Mixture-of-Experts large language model-based automatic speech recognition system that addresses real‑world production challenges such as regional dialects, dynamic entities, hotwords, long‑range context, and disfluent speech. Built on the Qwen backbone and trained on tens of millions of hours of speech data, it supports transcription in 30 languages and 16 Chinese dialects, and offers industry‑domain entity recognition, hierarchical hotword customization, single‑pass polishing, and long‑audio contextual modeling. A streaming variant, Qwen-Audio-3.0-ASR-Streaming, is also presented for low‑latency applications, with evaluations showing state‑of‑the‑art performance against leading commercial systems.

By Chuanmeng Bian, Daren Chen, Peixin Chen, Zhigao Chen, Zhiyun Fan, Zhifu Gao, Bo Gong, Qing Gu, Jiajun He, Yawei Hu, Yunjie Ji, Jingbei Li, Xiangang Li, Xu Li, Zengxi Li, Zheng Li, Chengdong Liang, Baiji Liu, Ying Liu, Bin Ma, Yiping Peng, Yuezhang Peng, Zhendong Peng, Yu Pu, Yang Shi, Xin Shu, Jian Tang, Biao Tian, Peiyao Wang, Tianzi Wang, Wen Wang, Wupeng Wang, Cheng Wen, Yuzhong Wu, Zijian Xia, Yunchong Xiao, Nan Yang, Jianwei Yu, Jixing Yu, Binbin Zhang, Lei Zhang, Sitong Zhao, Guangdong Zhou, Yuan Zhou, Jianheng Zhuo
arXiv Machine Learning
Sep 10

Selective Posterior Margin Regularization for Forward-Corrected Classification

The paper introduces Selective Posterior Margin Regularization (SPMR), a technique that enhances Forward correction for learning with class‑conditional label noise. SPMR preserves the Forward objective while converting disagreements between the corrected likelihood’s reverse posterior and the observed annotation into a graded update on the clean classifier. Experiments on five known‑transition benchmarks show that SPMR improves performance by 2.5–7.0 percentage points over full‑length Forward and remains 0.7–2.5 percentage points better when combined with Mixup and early stopping, with gains attributed to posterior‑space coefficients, transition‑adjusted targets, and pairwise actions.

By Zexing Zhang, Jichao Li, Tianyang Lei, XiongYi Lu, Yang Kewei
arXiv Machine Learning
Sep 10

IXPLORE: Bounded Ideal Point Estimation with Grid-Based Uncertainty Quantification

IXPLORE is a bounded ideal point estimation algorithm that blends a predictive fit objective with a sparsity‑aware likelihood function. It outperforms both traditional model‑based methods like IRT and other machine‑learning approaches on reconstruction and imputation error across five benchmark datasets, especially for users with sparse responses. The method also incorporates non‑linear feature transforms for further error reduction while maintaining visual interpretability, and it quantifies uncertainty via grid‑based posterior inference on a bounded 2D latent space.

By Fynn Bachmann
arXiv Machine Learning
Sep 10

Compressed Recurrent Feedback in Tsetlin Machines: A Reproducible Boolean-FSM Study

The paper proposes a fixed‑width recurrent feedback scheme for Recurrent Tsetlin Machines (RTMs) by folding clause activations with XOR, retaining the folded bits at two time scales, and thresholding them back to binary. This compression reduces 480 clause activations to 96 recurrent bits while achieving comparable accuracy (61.47 ± 6.74% and 62.94 ± 9.92%) on a reproducible Boolean finite‑state‑machine benchmark across 144 runs. The study shows that raw clause feedback offers only marginal accuracy gains but increases recurrent width and execution time significantly, and highlights the importance of no‑feedback controls in sequence model benchmarking.

By Ankit Kumar, Utkarsh Raj, Rishad Shafik, Sudip Roy
arXiv Machine Learning
Sep 10

Role-Specific Predictive Geometries for Nonstationary Multivariate Graph-Signal Forecasting

The paper introduces role‑specific predictive geometries for forecasting nonstationary multivariate graph signals. It distinguishes long‑run equilibrium restoration (Long relations) from short‑run transient propagation (Short relations), applying directed Long relations to equilibrium coordinates and directed Short relations to lagged differences. Experiments show perfect recovery of planted relations and improved performance over a G‑VARMA baseline on several real‑world benchmarks.

By Yanbo Chen, Anamitra Makur
arXiv Machine Learning
Sep 10

A Statistical and Machine Learning Framework for Quantifying Offensive Impact in Professional Box Lacrosse

The paper presents a statistical and machine‑learning framework for estimating expected goals (xG) and attributing offensive involvement in professional box lacrosse. Using 1,006 manually annotated shot attempts from 13 Rochester Knighthawks games, the authors evaluate logistic regression, random forest, and extremely randomized trees, finding that a contextual baseline random forest achieves the lowest log loss and Brier score. They also introduce a shot‑based expected assist metric and an exploratory expected pick value (xPV) metric, noting that xPV’s magnitude is indistinguishable from noise but shows a directional pattern in permutation tests.

By Robert Jimerson Jr
arXiv Machine Learning
Sep 10

AF-Mamba: Efficient Long-Term Signal Modeling for Early Prediction of Atrial Fibrillation Onset

AF-Mamba is a deep learning model that predicts atrial fibrillation (AF) onset one hour in advance using long‑term RR intervals. It combines temporal convolutional networks for local feature extraction with Mamba, a state‑space model for long‑range sequence modeling, achieving high sensitivity (0.889) and specificity (0.943) in subject‑wise testing. The model maintains strong performance across unseen datasets, offering a favorable trade‑off between predictive accuracy and computational efficiency for real‑time ambulatory monitoring.

By Yongbin Lee, Ki H. Chon
arXiv Machine Learning
Sep 10

TrojanWorld: Backdooring World-Model Agents via Imagination Steering

TrojanWorld is a backdoor framework that targets world-model agents by steering their internal imagination toward attacker-specified actions when a physical trigger is present. The attack uses Decision-Reflective Induction, Clean Behavior Anchoring, and Causal Propagation to maintain stealth, persistence, and high performance. Experiments on TD-MPC2, DreamerV3, and R2-Dreamer across several benchmarks show that the attack can induce target actions with minimal performance loss and can keep agents on a malicious trajectory even after the trigger is removed.

By Wenkai Huang, Siyuan Liang, Gaolei Li, Yiming Li, Tianhao Peng, Jianhua Li, Dacheng Tao
arXiv Machine Learning
Sep 10

Trust-But-Verify: Poisoning-Resilient Locally Private Graph Learning Protocols

The paper introduces VERITAS, a poisoning‑resilient protocol for locally private graph learning that combines local differential privacy with a trust‑but‑verify approach. VERITAS employs a verification list to encode peer trust, performs local data perturbation, server‑side malicious node pruning, dual denoising, and robust private graph learning. Experiments on four real‑world datasets show that VERITAS defends against data poisoning attacks while improving downstream graph learning utility under strict privacy guarantees.

By Longzhu He, Li Sun, Hao Peng, Ruijie Wang, Raymond Chi-Wing Wong, Sen Su
arXiv Machine Learning
Sep 10

On the Recall Scaling Laws in Mamba: A Theoretical and Mechanistic Study via Hashing

The paper investigates Associative Recall (AR) in the Mamba architecture, showing that Mamba implicitly learns linear hash functions to perform recall. It identifies the low‑level circuit responsible for this behavior and develops a theoretical framework—Recall Scaling Laws—based on similarity‑preserving hashing principles. The framework predicts embedding and state dimensions for perfect recall, recall success probability, and analyzes multi‑layer and multi‑head SSM patterns, with empirical results confirming its accuracy.

By Yuval Koren, Assaf Ben-Kish, Raja Giryes, Lior Wolf, Itamar Zimerman
arXiv Machine Learning
Sep 10

ParetoTransport: Generative Optimization by Mass Transport Toward The Pareto Front

ParetoTransport is a training‑free guidance method for pre‑trained flow‑matching models that explicitly refines a population‑level distribution in objective space. It iteratively transports the empirical offline distribution toward the Pareto front using Wasserstein matching to intermediate proxy distributions, thereby controlling distributional displacement and mass allocation along the front. The authors prove a convergence result and show state‑of‑the‑art performance on standard offline multi‑objective optimization benchmarks, evaluating beyond hypervolume to generational distance, inverted generational distance, and Wasserstein distance.

By Stephanie Holly, Sepp Hochreiter, Werner Zellinger
arXiv Machine Learning
Sep 10

Heat Field Signatures: From Point Clouds to Smooth Geometry

Heat Field Signatures (HFS) lift irregular point clouds into a multiscale family of smooth ambient heat fields, enabling closed‑form computation of global and local geometric signatures directly from pairwise distances. HFS captures heat concentration, intrinsic dimension, anisotropy, and scale transitions, and introduces the Heat Dimension Spectrum (HDS) as a compact multiscale summary. The method serves as a descriptor, lightweight learned representation, or feature channel for neural point‑cloud models, outperforming strong baselines on synthetic and real‑world benchmarks while reducing end‑to‑end cost.

By Yuanqing Wang, Yapeng Tian, Baris Coskunuzer
arXiv Machine Learning
Sep 10

Online Signature Verification Using Augmented Path Signature and T-Mamba

The paper introduces a new online signature verification framework that combines the augmented path signature (APS) descriptor with a T-Mamba model. APS applies time and basepoint augmentations followed by sliding-window path signatures, capturing geometric structures and nonlinear inter-channel interactions. T-Mamba, a hybrid of two temporal convolutional network blocks and a time-scanning Mamba, learns both local temporal patterns and global long-range dependencies, achieving state‑of‑the‑art equal error rates on three public benchmark datasets.

By Ruiling Li, Danyu Yang