Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,322 stories · RSS feed

arXiv AI
Sep 11

XAI-Arena: Can LLMs Assess the Quality of XAI Explanations?

XAI-Arena proposes using large language models (LLMs) as judges to evaluate the quality of explainable AI (XAI) explanations, aiming for reproducibility, scalability, and multidimensional assessment. The framework assesses dimensions such as simplicity, clarity, task adequacy, trust calibration, actionability, transparency, faithfulness, and overall interpretability across different datasets, models, and stakeholder personas. Human validation shows a strong positive correlation between LLM-generated and human ratings (Spearman's rho = .693, p < .001), supporting the viability of LLM-based evaluations.

By Yanfei Hu Fleischhauer, Alona Zharova, Nadja Klein, Stefan Feuerriegel
arXiv AI
Sep 11

Complementing reinforcement learning with SFT through logit averaging in the post training of LLMs

The paper proposes a new technique that averages the logits of a frozen reference policy (such as one obtained via supervised fine‑tuning) and a trainable policy, integrating this approach into Group Relative Policy Optimization (GRPO). Unlike methods that use Kullback–Leibler regularization or a critic, the proposed method couples the policies solely through logit averaging, aiming to combine the reasoning strengths of the trainable policy with the formatting benefits of the reference policy. Experiments on MATH, cn‑k12, and MMLU demonstrate that this approach achieves higher or comparable accuracy to the standard KL‑regularized GRPO.

By Xingwei Gan, Ying Zhu
arXiv Machine Learning
Sep 11

M3-Former: Multimodal Transformer with Mixture-of-Experts for Long-Term Vessel Trajectory Prediction

M3-Former is a multimodal transformer framework that uses large language models to encode vessel static attributes and navigational intent as semantic priors for long‑term trajectory prediction. It builds a unified multimodal representation space, aligns static semantic information with dynamic trajectory features via self‑attention, and employs a dual‑granularity Mixture‑of‑Experts architecture to capture both global route planning and fine‑grained maneuvering behaviors. A Steering‑Weighted Cross‑Entropy loss further improves accuracy on sparse turning events, and experiments on a Danish AIS dataset show consistent improvements over state‑of‑the‑art baselines, reducing ADE and FDE by up to 5.1% in 4‑hour predictions.

By Wenzhe Jin, Haina Tang
arXiv Machine Learning
Sep 11

Halo: Improving forecast accuracy through heteroscedastic estimation

Halo is a modification to existing deep forecasters that adds a second output for estimating the scale of the predicted distribution, trained with a matching negative log likelihood. Experiments on five electricity price markets show that Halo improves mean squared error and mean absolute error in 28 of 30 model‑market‑metric comparisons, with average MSE reductions of 2.6% to 16.5% and MAE reductions of 1.7% to 11.0%. The study finds that the source of the scale estimate is less important than the fact that the network estimates scale, and that the improvement persists without retuning hyperparameters.

By Adam Cataldo
arXiv Machine Learning
Sep 11

RDDMPI: Residual Denoising Diffusion Model for Probabilistic Multivariate Time Series Imputation

RDDMPI introduces a residual denoising diffusion model for multivariate time series imputation. By decomposing the missing signal into a baseline reconstruction and a residual uncertainty component, the method conditions the diffusion process on both the completed signal and its latent representation, using a reliability-aware mechanism to balance baseline influence. Experiments on benchmark datasets show that this approach improves reconstruction accuracy and uncertainty quantification compared to prior diffusion-based methods.

By Ramiro Valdes Jara, David Chapman, Adam Meyers
arXiv Machine Learning
Sep 11

Learning structural balance of graphs from quantum spectral features

The paper introduces a quantum method for extracting spectral features from the density of states (DOS) of a problem-dependent Hamiltonian, applied to signed graphs represented as Ising models. Using standardized moments of the Ising DOS as features, the authors demonstrate that these moments count signed closed walks, are switching‑invariant, and size‑free. On a benchmark of 140,000 labeled graphs, the exact DOS predicts the frustration index exactly, while five moments achieve a mean error of 0.4, and a new DOS‑QPE protocol offers efficient sampling with far fewer shots than classical trace‑sampling methods.

By Stefano Scali, Oleksandr Kyriienko
arXiv Machine Learning
Sep 11

Solving Few-Shot Multiobjective Multitask Optimization via Iterative Sequential Transfer

The paper introduces Iterative Sequential Transfer (IST), a method for few-shot multiobjective multitask optimization that addresses the challenge of aligning elite solution distributions across tasks. IST treats multitask optimization as a sequence of transfer problems, focusing evaluations on one target task per iteration and using a likelihood-informed prioritization to select the task most ready for knowledge integration. Experiments on benchmark and real-world problems demonstrate the method’s effectiveness under tight evaluation budgets.

By Tingyang Wei, Haofeng Wu, Ananda Phan Iman, Zhao Wei, Jiao Liu, Yew-Soon Ong
arXiv Computation and Language
Sep 11

BodyCam-VQA: Enhanced Body-Worn Camera Video Captioning via Multimodal Reasoning and Probe Question Generation

The paper introduces BodyCam-VQA, an adaptive visual question answering framework designed to improve captioning of police body‑worn camera footage. By employing structured multimodal reasoning and probe question generation, the system extracts fine‑grained visual evidence that conventional captioning models miss. Experiments with various question generation models show that this VQA‑driven approach yields more reliable, objective, and detailed records of enforcement events.

By Karish Gupta, Matthew Alex, Alex Li, Yang Wu, Yun-Wei Chu, Kashif Munir, Xiaotian Zhou, Zhengping Ji, Xiaozhong Liu
arXiv Computer Vision
Sep 11

Beyond Visual Quality: Evaluating Physical Consistency under Ego-Motion with EgoGenEval

The paper introduces EgoGenEval, a new benchmark that assesses the physical consistency of visual generators under ego‑motion by measuring Camera Motion Grounding and Scene State Preservation across 1,400 cases and 2,360 target views. Experiments on 16 pose‑free generators and two pose‑conditioned references show that current models struggle to maintain both camera motion and scene state simultaneously. A follow‑up study using EgoGen‑Train demonstrates that pairwise supervision does not effectively improve both metrics together, suggesting the need for a trajectory‑centric training paradigm.

By Yilin Long, Chenming Zhu, Zitang Gou, Jingli Lin, Tai Wang
arXiv Computer Vision
Sep 11

Harnessing Intrinsic Subject-Aware Attention for Controllable Multi-Subject Video Generation

The paper tackles two main issues in multi‑subject video generation—uncontrollable fidelity strength and semantic drift—by exploiting intrinsic attention patterns in Diffusion Transformers. It introduces an Intrinsic Spatial Grounding Map (ISGM) that accurately locates reference subjects and a Dual‑phase Intrinsic Attention Leveraging (DIAL) framework that uses ISGM during both training and inference. DIAL guides attention in low‑noise stages for precise fidelity control and builds preference pairs in high‑noise stages for reinforcement learning, resulting in superior identity consistency and controllable fidelity on the OpenS2V‑Eval benchmark.

By Niange Yu, Ye Tian, Biaolong Chen, Miao Lu, Aixi Zhang, Hao Jiang, Yunhai Tong, Pipei Huang
arXiv Computer Vision
Sep 11

Learn the Solid, Not the File: Canonical Inputs for Neural Networks on CAD Boundary Representations

The paper examines how boundary representation (B‑rep) encoders for CAD models fail to handle variations that do not change the underlying solid, such as different modeling operations or software export settings. By demonstrating that existing encoders collapse under these perturbations, the authors introduce the canonical region graph, a representation derived directly from the solid that is theoretically invariant to repartitioning and rigid motions. This new input format matches the best baseline on standard benchmarks and remains stable across all tested perturbations.

By Heinrich Jiang, Hager Yasser Mohamed, Alexander Hitt, Valeriia Lomakina, Henning Jiang, Jennifer Jang
arXiv Computation and Language
Sep 11

Beyond Word Error Rate: A Switch Aware Evaluation of ASR and Audio Language Models on English Yoruba Code-Switched Speech

arXiv:2609. 11786v1 Announce Type: new Abstract: Automatic speech recognition (ASR) systems and audio language models (audio LMs) now report low error rates on monolingual benchmarks, but their behavior on code switched speech in low resource, diacritic rich languages remains poorly characterized.

By Chibuzor Okocha, Christan Earl Grant
arXiv Computation and Language
Sep 11

Biology-in-the-loop: Amortized Adaptive Hit Discovery in CRISPR Screens

The paper introduces AssayBench-Loop, a large benchmark of 1,389 CRISPR screens across five phenotype categories, and builds on it to develop AssayLoop, a sequential experimental design framework that combines a transformer-based acquisition policy (AssayFormer) trained on historical data with LLM-derived biological priors. AssayLoop achieves a 5.67‑fold enrichment over random selection, recovering 27.7% of hits after testing only about 5% of the library, and outperforms existing adaptive-design methods and standalone LLMs. The authors also present AssayLLM, extending the approach directly to an LLM via task‑specific post‑training, and show that performance improves with more historical training data and transfers to unseen phenotype categories.

By Carl Edwards, Edward De Brouwer, Xiner Li, Namkyeong Lee, Ehsan Hajiramezanali, Anne Biton, Sara Mostafavi, Gabriele Scalia
arXiv Computation and Language
Sep 11

Analyzing LLM Reasoning to Uncover Mental Health Stigma

The paper investigates how large language models (LLMs) can exhibit stigma toward people with psychological conditions by examining their intermediate reasoning steps rather than just final answers. Using clinical expertise, the authors develop a framework to identify and rate stigmatizing language in LLM reasoning, distinguishing between overt prejudice and subtler biases. They also expand an existing mental health stigma benchmark to include more psychological conditions, finding that reasoning analysis reveals far more stigma than traditional multiple-choice evaluations and exposes flaws in the models’ logic and understanding of mental health.

By Sreehari Sankar, Aliakbar Nafar, Mona Barman, Hannah K. Heitz, Ashwin Kumar, Pouria Tohidi, Dailun Li, Danish Hussain, Russell DuBois, Hamed Hasheminia, Farshad Majzoubi
arXiv Computer Vision
Sep 11

SAMV-DUSt3R: Instance-Centric 3D Scene Decoupling from Sparse Multi-Views

SAMV-DUSt3R is an end‑to‑end model that injects SAM2 2D masks into MV‑DUSt3R reconstruction to decouple objects from 3D scenes. A Cross Flow Mask Block steers the network toward the target instance, improving shape accuracy and achieving object‑level disentanglement without multi‑stage pipelines. A lightweight Spatial RankGNN selects the optimal reference view with 73.5% accuracy, and experiments show an 11% boost in average reconstruction precision over state‑of‑the‑art baselines.

By Langxu Zhao, Zuan Gu, Yingdan Zhang, Pengfei Zhao, Tianhan Gao
arXiv Computer Vision
Sep 11

BruNet: A Cross-Domain Transfer Framework for Bruise Segmentation

BruNet is a new cross‑domain transfer framework for automatic bruise segmentation that combines a ViT‑based visual encoder (either self‑supervised DINOv3 or pretrained LingBot‑Vision) with a SAM‑based mask decoder. The model is trained on the HAM10000 skin lesion dataset and evaluated on a separate bruise dataset without any fine‑tuning, achieving superior performance over CNN‑based models, state‑of‑the‑art segmentation models, ChatGPT‑4o/5‑assisted SAM2 zero‑shot baselines, and the medical‑oriented MedSAM. This work represents the first study to address pixel‑level localisation of bruises, demonstrating strong cross‑domain generalisation.

By Qiming Wang, Richard J. Motley, Ebube E. Obi, Xianfang Sun, Paul L. Rosin
arXiv Computer Vision
Sep 11

Single-Stream Multi-Feature Fusion with Temporal Robustness for Gait Emotion Recognition

The paper introduces SV-GCN, a single-stream multi-feature fusion framework for 3D skeleton-based gait emotion recognition that incorporates temporal invariance. It uses intra-frame relative motion features to remove frame-rate sensitivity and embeds heterogeneous cues at shallow layers for early fusion, avoiding multi-stream complexity. A global mask-guided valid-frame spatio-temporal graph convolution module further enhances robustness to variable-length sequences and differing frame rates, achieving state‑of‑the‑art performance on the E‑Gait dataset and strong generalization across sequence lengths.

By Shirong Lyu, Silu Quan, Yixuan Ding, Chengpeng Wang
arXiv Computer Vision
Sep 11

Leveraging Avatar Fingerprinting: A Multi-Generator Photorealistic Talking-Head Public Database and Benchmark

The paper introduces AVAPrintDB, a new public multi‑generator talking‑head avatar database designed for avatar fingerprinting, comprising data from two audiovisual corpora and three state‑of‑the‑art generators (GAGAvatar, LivePortrait, HunyuanPortrait). It also defines a standardized benchmark that evaluates existing avatar fingerprinting systems and explores new methods based on Foundation Models such as DINOv2 and CLIP, while analyzing performance under generator and dataset shift. The authors find that identity‑related motion cues persist across synthetic avatars, yet current fingerprinting systems are highly sensitive to changes in synthesis pipelines and source domains.

By Laura Pedrouzo-Rodriguez, Luis F. Gomez, Ruben Tolosana, Ruben Vera-Rodriguez, Roberto Daza, Aythami Morales, Julian Fierrez
arXiv Computation and Language
Sep 11

SearchAtlas: Analyzing Agentic Search Strategies via Evidential Query Graphs

SearchAtlas is a framework that transforms raw search trajectories of large language model (LLM) agents into structured evidential query graphs, where edges capture how evidence is propagated from queries to the final answer. The automated parsing pipeline achieves a mean edge F1 of 86.0% against human-annotated graphs and remains consistent across repeated runs. Using SearchAtlas, the authors analyze five search agents on three benchmarks, uncovering systematic differences in search scale and evidence aggregation, and revealing process failures such as fragmented answer support, unmet question constraints, and unverified parametric knowledge that correlate strongly with incorrect answers.

By Jiacheng Sang, Mengyuan Li, Sanxing Chen, Yukun Huang, Yu Feng, Bhuwan Dhingra
arXiv Computation and Language
Sep 11

ReGround: Grounding Reviewer Comments in Multimodal Evidence

ReGround is a new large‑scale dataset that links 10,267 reviewer comments to 16,274 pieces of evidence across 3,656 anonymous scientific papers, addressing the challenge of grounding comments in long multimodal documents. The dataset is constructed by leveraging explicit references in author rebuttals, providing high‑precision annotations. Evaluation shows that simple retrieval over full paper text performs poorly, evidence‑type inference is a major bottleneck, and multimodal evidence offers complementary signals that pure text misses.

By Serwar Basch, Lizhen Qu, Iryna Gurevych