Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,322 stories · RSS feed

arXiv Machine Learning
Sep 22

Temporal Generalization and Explanation Stability of Control Flow Graph Neural Networks for Malware Detection

The paper evaluates how graph neural networks (GNNs) built on control flow graphs (CFGs) perform when trained on one time period and tested on a later one, using a strict temporal split. Twelve GNN variants and a flat-feature baseline were trained on 459 CFGs from 2024‑2025 and evaluated on 223 CFGs from 2026, revealing that the choice of message‑passing operator strongly affects robustness to distribution shift. Attribution stability varied by architecture, and the best-performing operator on the later corpus was also the hardest to explain, leading the authors to design a new architecture that matches its performance without search.

By Md. Asif Sajeed, Md. Nazrul Islam Mondal, Md Ashraful Hossen Akash
arXiv Computation and Language
Sep 22

MechaTerp-TRACE: A Novel Approach for Component Ablation Analysis in Language Models

MechaTerp-TRACE is a new framework that systematically ablates individual components of language models to measure their causal contribution to producing a named entity. By applying TRACE to thirteen instruction‑tuned dense decoder models, the study finds that a small set of positionally fixed components consistently carry the most influence across models and prompts, while the remaining support is evenly distributed. This suggests that entity knowledge is largely embedded in generic generation machinery rather than in isolated, findable components.

By Brandon Colelough, Davis Bartels, Madeline Bittner, Dina Demner-Fushman
arXiv Computation and Language
Sep 22

Cross-Dialect NER for Bangla Regional Dialects Using Leave-One-Dialect-Out Cross-Validation and Explainable AI

The paper introduces a cross-dialect Named Entity Recognition (NER) framework for Bangla, leveraging the ANCHOLIK-NER dataset that covers five major regional dialects. Using a Leave-One-Dialect-Out Cross-Validation strategy, eight transformer-based models were evaluated, with Multilingual-E5 Large achieving the best performance (F1 up to 97.26% on Mymensingh, 82.38% on Chattogram). Local Interpretable Model-agnostic Explanations (LIME) revealed that the models rely mainly on the surface form of entity words rather than surrounding context, suggesting a direction for future improvement.

By Shamim Rahim Refat, Faika Fairuj Preotee, Shuvashis Sarker, Shifat Islam, Bidyarthi Paul, Mohammad Ashraful Hoque
arXiv Computer Vision
Sep 22

MinCU: A Fine-Grained Benchmark for Grounded Minimal-Change Understanding in Image Pairs

MinCU is a new benchmark for grounded minimal‑change understanding that presents pairs of near‑identical images differing by a single atomic variation in object category, attribute, count, or spatial position. Models are evaluated on their ability to describe the change, localize the changed region, and identify the changed entity. The authors also introduce SG‑ISA, a structured autoregressive method that decomposes the task into a Think‑Locate‑Describe sequence, showing that fine‑tuning with SG‑ISA improves both grounding accuracy and description quality while reducing reasoning‑token overhead.

By Chaoqian Mu, Wenhao Wu, Zichen Liang, Jiaxu Li, Lijun Wang, Yifan Wang, Huchuan Lu
arXiv Computer Vision
Sep 22

Comparative Performance and Parameter-Efficient Adaptation of DINOv2 for Active Trachoma Classification

The study evaluates the performance of the DINOv2 visual representation for classifying Trachomatous Inflammation-Follicular (TF) versus normal conjunctival images. Using 1,546 images processed by the OPTED pipeline, the authors compare six pretrained backbones and then test four lightweight adaptation methods on DINOv2 ViT-B/14. The best results—91.66% accuracy, 90.69% macro‑F1, and 96.06% AUC—were achieved with DINOv2 plus Efficient Channel Attention (ECA) and a focal‑plus‑center loss, though ECA’s benefit varied with the loss function.

By Kibrom Gebremedhin, Hadush Hailu, Bruk Gebregziabher, Yordanos Hailu
arXiv Computer Vision
Sep 22

WebMRIQC: A Web-Based Implementation of MRIQC for Accessible MRI Image Quality Assessment in Resource-Constrained Settings

WebMRIQC is a browser-based, open‑source platform that wraps the MRIQC engine to provide automated MRI image quality assessment without local installation. It handles DICOM‑to‑BIDS conversion, runs the containerized MRIQC pipeline on a shared compute node, and presents results in an interactive dashboard that benchmarks each scan against normative data. Validation on BraTS‑Africa and BraTS 2021 datasets shows strong agreement across thirteen image‑quality metrics, indicating that the web implementation can match native MRIQC performance.

By Philip Nkwam, Ifeoluwa Oladeji, Sekinat Zurakat-Aderibigbe, Jasmine Cakmak, Harrison Aduluwa, Confidence Raymond, Cliff Mokua, Abdulrazaq Zubair, Daniel Champanda, Tolulope Olusuyi, Maruf Adewole, Udunna Anazodo
arXiv Computer Vision
Sep 22

AlignMorph: Tuning-Free Diffusion Image Morphing via Explicit Semantic Transport

AlignMorph is a tuning‑free diffusion framework for image morphing that separates geometric alignment from generative denoising. It uses Global Semantic Transport—entropic optimal transport and reliability‑aware latent warping—to achieve diffusion‑compatible semantic alignment, and Coordinate‑Aligned Generation—symmetric bi‑phase attention handoff—to preserve spatial coordinates during denoising. The method eliminates ghosting and delivers superior structural coherence and temporal smoothness on morphing benchmarks without any per‑pair optimization.

By Wuyi Liu, Xu Han, Yuren Chen, Yige Mao, Zishuo Peng, Xianzhi Li
arXiv Computer Vision
Sep 22

LIBERO-VPro: Benchmarking Closed-Loop Visual Robustness of Robotic Foundation Models

LIBERO-VPro is a benchmark designed to assess the closed‑loop visual robustness of robotic foundation models by systematically perturbing visual inputs during task execution. It spans four dimensions—Visual Evidence Degradation, Camera Staleness, Visual Source Consistency, and Task‑Relevant Scene Variation—across 12 challenge categories, 96 settings, and 3,296 task‑condition cases. Evaluations on six models over 196,000 simulated episodes and 200 real‑world rollouts show that high nominal performance can hide significant weaknesses in visual grounding, adaptation, and sensitivity to stale or missing observations.

By Huiqiong Li, Zhiting Mei, Anirudha Majumdar, Jingjing Chen, Yu-Gang Jiang, Bin Zhu
arXiv Computer Vision
Sep 22

Closed-Circuit Television Data as an Emergent Data Source for Urban Rail Platform Crowding Estimation

The paper explores the use of Closed‑Circuit Television (CCTV) footage to estimate urban rail platform crowding in real time. It compares three computer‑vision methods—object detection and counting, crowd‑level classification with a Vision Transformer, and semantic segmentation—to extract crowd-related features. A novel convex ridge regression technique is introduced to convert segmentation outputs into passenger counts, and the methods are evaluated on a privacy‑preserving dataset of over 600 hours of Washington Metropolitan Area Transit Authority (WMATA) video, showing that CCTV alone can provide valuable real‑time crowd estimates.

By Riccardo Fiorista, Awad Abdelhalim, Anson F. Stewart, Gabriel L. Pincus, Ian Thistle, Jinhua Zhao
arXiv Computer Vision
Sep 22

Mitigating Domain Shift in Conditioned Floor Plan Generation: Synthetic Pre-training for Data-Efficient Adaptation

The paper investigates how conditioned floor plan generation models perform when applied to datasets from different regions, revealing significant performance drops due to domain shift. To address this, the authors create a large synthetic training set that enforces physical constraints while deliberately reducing architectural realism, and show that pre‑training on this data boosts zero‑shot cross‑domain performance and speeds up fine‑tuning in low‑data scenarios.

By Matthieu Ospici, Arnaud Gueze, Luc Bourrat, Adrien Bernhardt
arXiv Machine Learning
Sep 22

The Effect of Quantization on Clinical Benchmarks: Accuracy and Safety Across Model Families

The study evaluates how quantization affects accuracy and safety of five 7‑8B language models on clinical benchmarks. INT8 GPTQ shows minimal degradation (≤1.9%) across tasks, while INT4 causes substantial, model‑dependent drops, especially in high‑risk scenarios and safety metrics. Recovery methods such as clinical calibration substitution and QLoRA fine‑tuning yield mixed results, underscoring the need for task‑specific validation.

By Leonard Twagirayezu, Prasenjit Mitra
arXiv Machine Learning
Sep 22

Muon Can Outperform Dedicated Continual Learning Methods

arXiv:2609.24678v1 Announce Type: new Abstract: Continual learning with Low-Rank Adapters (LoRA) typically mitigates forgetting by penalizing the overlap between a new update and the accumulated past...

By Sebastian George Sincari (Faculty of Mathematics and Computer Science, University of Bucharest, Bucharest, Romania), Bogdan Alexandru Gheorghe (Faculty of Mathematics and Computer Science, University of Bucharest, Bucharest, Romania), Antonio Barbalau (Bitdefender, Bucharest, Romania)