Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,147 stories · RSS feed

arXiv Machine Learning
Oct 3

Explainable Suicide Risk Assessment on Social Media with Multi-Task QLoRA

The paper presents a system for the IEEE BigData 2026 Cup on Explainable Suicide Risk Assessment on Social Media. It tackles three tasks—risk-level classification, evidence phrase extraction, and multi-label factor identification—using Qwen2.5-Instruct models adapted with quantized low-rank adaptation (QLoRA) and an answer-masked causal language-model objective. The final system achieved a composite score of 0.7738, with 0.8089 on Task 1 and 0.6919 on Task 2, demonstrating that task‑specific training and tailored aggregation improve performance across the three tasks.

By Xuan Zhong Feng, Geoffrey Martin, Hexin Dong, Yifan Peng
arXiv AI
Oct 2

HydroJEV: A one-second, training-free screen for cyber-attack and fault attribution in water distribution networks

HydroJEV is a training‑free, one‑second model that classifies SCADA alarms in water distribution networks into cyberattack, physical fault, normal transient, or faulty sensor. In a benchmark on the C‑Town EPANET network, HydroJEV matched a hand‑written rule tree and outperformed a supervised classifier, especially when few labeled events were available, while being 20‑40 times faster than large language models. When combined with a rule tree gate, it reduced the need for human review by about a third without sacrificing accuracy.

By Tianwei Mu, Shengyan Jiang, Mingzhe Yuan, Qing Luo, Min Xiao, Wenhong Wang, Jun Li, Manhong Huang
arXiv AI
Oct 2

PACE: Provenance-Aware Capability Enforcement for Tool-Using LLM Agents

The paper introduces PACE, a Provenance-Aware Capability Enforcement system designed to secure tool-using large language model agents by mediating every tool call before execution. PACE employs path confinement to limit influence paths and verifies effects against authenticated authority, distinguishing certified execution contracts from evaluated configurations. Experiments on eight agent‑security benchmarks show that the evaluated configuration reduces attack success in most cases while maintaining near‑native utility.

By Fengpeng Li, Qizhou Wang, Yuke Hu, Kemou Li, Jun Liu, Haiwei Wu, Jiantao Zhou, Di Wang
arXiv AI
Oct 2

CODesign: Consistency from Data to Trajectory in All-Atom Protein Binder Co-Design

CODesign is a co-design framework that jointly generates protein sequences and structures to improve consistency between them. It introduces a large consistency‑distilled dataset of about 105,000 dimers and employs a multimodal joint flow model with a consistency‑aware resampling strategy to iteratively refine sequences and side chains. The approach achieves state‑of‑the‑art in silico success rates for protein‑ and ligand‑target binder design, with ablation studies showing a 70.9% performance boost from the distilled dataset and further gains from the resampling mechanism.

By Yuanle Mo, Bo Qiang, Haitao Lin, Qinghan Wang, Gang Du, Odin Zhang, Pheng Ann Heng
arXiv AI
Oct 2

ProtoDCS: Towards Robust and Efficient Open-Set Test-Time Adaptation for Vision-Language Models

ProtoDCS introduces a robust open‑set test‑time adaptation framework for vision‑language models, addressing the challenge of simultaneously handling covariate‑shifted in‑distribution (csID) and out‑of‑distribution (csOOD) data. It replaces brittle thresholding with a double‑check separation using a probabilistic Gaussian Mixture Model and employs an evidence‑driven adaptation strategy that updates prototypes efficiently, reducing overconfidence and computational cost. Experiments on CIFAR‑10/100‑C and Tiny‑ImageNet‑C show state‑of‑the‑art performance, improving both known‑class accuracy and OOD detection metrics.

By Wei Luo, Yangfan Ou, Jin Deng, Zeshuai Deng, Xiquan Yan, Zhiquan Wen, Mingkui Tan
arXiv Machine Learning
Oct 2

RW-Flow: One-Step Generation on Compact Manifolds via Wasserstein Gradient Flows

RW-Flow presents a new one‑step generative framework for data on compact manifolds, leveraging Wasserstein gradient flows. The authors derive a necessary and sufficient identifiability condition for velocity fields on compact, connected Riemannian manifolds, showing that a symmetric, Lipschitz‑continuous cost function yields identifiability iff its Gibbs kernel is nondegenerate. Experiments on geospatial events, protein and RNA torsion angles, and discretized manifolds demonstrate that RW‑Flow surpasses existing one‑step methods across most benchmark settings.

By Ualibyek Nurgulan, Seungwoo Yoo, Prin Phunyaphibarn, Minhyuk Sung
arXiv Machine Learning
Oct 2

Accelerated Algorithm for Sparse Regularized Partial Optimal Transport

The paper introduces an accelerated algorithm for sparse regularized partial optimal transport (POT). It reformulates the problem using penalty-based techniques that allow efficient gradient updates while maintaining the original structure, and supports a wide range of regularizers that encourage structured sparsity. Experiments on color transfer, domain adaptation, and point cloud registration show the method outperforms existing baselines in transport cost, sparsity, and convergence speed.

By Khoa Nguyen, Dung T. Nguyen, Thong Huynh, Hoang-Hiep Nguyen-Mau, Anh Nguyen, Minh Ngoc Dinh, Juho Kannala
arXiv Machine Learning
Oct 2

JET: Justification Evaluation in Transformer

JET (Justification Evaluation in Transformer) leverages pretrained language and vision‑language models to choose among a limited set of answers without extra training. It directly evaluates candidate likelihoods, reuses computation across candidates, and runs experiments on desktop CPUs and consumer GPUs to measure decision accuracy and execution cost. Results show high accuracy on the MMLU test set, significant speedups from prefix reuse and cache management, and a 30.8% reduction in process time through input preparation optimizations, all while maintaining unchanged outputs.

By Shenghao Ding
arXiv Machine Learning
Oct 2

Let CSP Be Your ANCHOR: Adaptive Crystal Search over Frozen Structure Priors

The paper proposes ANCHOR, a composition policy that decouples crystal structure prediction (CSP) from de novo crystal generation (DNG) by using a frozen CSP model as a fixed evaluator and training the policy with multi‑objective rewards, including adaptive novelty. ANCHOR significantly improves metrics such as MSUN and SUN compared to traditional DNG approaches, and demonstrates transferability across different CSP backbones and distillation into Crystalite‑CSP. The study shows that fine‑tuning DNG models directly on these rewards yields limited gains, whereas the ANCHOR framework better leverages the CSP prior for more efficient crystal discovery.

By Emma Lei Hovmand, Jonas Elsborg, Melih Kandemir, Arghya Bhowmik
arXiv Computer Vision
Oct 2

Concept Driven Domain Adaptation: Finding an Abstract Needle in a Haystack

The paper introduces Concept-Driven Domain Adaptation (CDDA), a three-stage framework that adapts vision‑language models for concept‑to‑example video retrieval in educational settings. CDDA first structures textual embeddings using textbook and teacher‑handbook concept pairs, then transfers this geometry to documentary visuals with a frozen visual encoder, and finally jointly fine‑tunes both encoders with sparse visual concept supervision. On a middle‑school physics benchmark, CDDA outperforms several multimodal baselines in retrieving concept‑driven moments while preserving concrete image‑text alignment.

By Haiming Zhao, Tai Wang, Kun Zhang, Xicheng Peng, Zhiyang Li
arXiv Statistics ML
Oct 2

Roto-translated Local Coordinate Frames For Interacting Dynamical Systems

The paper introduces local coordinate frames for each node in geometric graphs to enforce roto‑translation (Galilean) invariance in modeling interacting dynamical systems. By aligning each node’s frame with its local geometry, the method enables anisotropic filtering in graph neural networks and improves learning of highly non‑linear, time‑dependent behaviors. Experiments on traffic scenes, 3D motion capture, and colliding particles show that this approach outperforms recent state‑of‑the‑art methods.

By Miltiadis Kofinas, Naveen Shankar Nagaraja, Efstratios Gavves
arXiv Statistics ML
Oct 2

Efficient Solvers for SLOPE in R, Python, Julia, and C++

The paper introduces a set of software packages in R, Python, Julia, and C++ that solve the Sorted L-One Penalized Estimation (SLOPE) problem efficiently. The packages employ a hybrid coordinate descent algorithm capable of fitting generalized linear models with various loss functions such as Gaussian, binomial, Poisson, and multinomial logistic regression. They support dense, sparse, and out‑of‑memory data structures, can compute the full SLOPE path, perform cross‑validation (including relaxed SLOPE), and are shown to outperform existing SLOPE implementations in speed on both real and simulated data.

By Johan Larsson, Malgorzata Bogdan, Krystyna Grzesiak, Mathurin Massias, Jonas Wallin
arXiv AI
Oct 2

A rubric landscape for evaluating clinical reasoning in large language models: what exists, what is missing, and what needs to be combined

The article reviews existing tools for assessing clinical reasoning in large language models, identifying six key dimensions—problem representation, temporal synthesis, differential and management reasoning, counterfactual reasoning, calibrated uncertainty, and reasoning faithfulness. It finds that no single instrument covers all dimensions, with some aspects reasonably addressed while others, especially faithfulness and counterfactual reasoning, remain weakly evaluated. The authors propose combining current tools through binary rubric items, separate completeness and correctness scores, importance weighting, safety caps, temporal consistency checks, and chance‑corrected reliability reporting, and call for further design work on uncertainty, counterfactual reasoning, and faithfulness in longitudinal free‑text contexts.

By Zhangshu Joshua Jiang, Zina Ibrahim, James T. Teo
arXiv AI
Oct 2

MCIR: A Feature Dependence-Aware Explainability Method with Reliability Guarantees

The paper introduces MCIR-M, a global feature‑importance method that accounts for feature dependence by conditioning each feature on its strongly dependent neighbours and computing a normalized Mutual Correlation Impact Ratio. MCIR-M yields a population score between 0 and 1, reaching zero when a feature is exactly conditionally redundant, and can be estimated with only a fraction of the data. Experiments on synthetic redundancy setups and the UCI HAR benchmark show that MCIR-M produces more stable rankings under near‑duplicate predictors compared to existing methods such as SHAP, SAGE, HSIC, and MI‑based scores.

By Poushali Sengupta, Sabita Maharjan, Frank Eliassen, Shashi Raj Pandey, Yan Zhang
arXiv AI
Oct 2

From Isolated Feature to Orbits: Discovering Music Concepts via Multi-SAE Alignment

The paper proposes a new interpretability framework for music foundation models that moves beyond isolated feature analysis to uncover structured relations, such as chords and keys, as organized orbits in pitch space. By generating pitch-shifted input pairs and aligning their Sparse Autoencoder (SAE) representations, the method identifies groups of pitch-related features that form these orbits. Experiments on two state‑of‑the‑art music models demonstrate that this approach can recover chord, key, and melodic pattern structures with minimal grounding examples.

By Liwei Lin, Gus Xia
arXiv AI
Oct 2

Mem++: Non-Destructive Memory for Long-Term Organizational LLM Agents

Mem++ is a non‑destructive memory framework for large language model agents that records every document in full, along with its date and author, instead of compressing it at write time. At read time, it retrieves only documents up to the requested time and fuses lexical and semantic rankings, leaving the choice of which version to use to the answering model. Evaluations on OrgMemBench show Mem++ outperforms the strongest baseline by 8.0 to 13.1 points and achieves the best overall score with gpt‑4.1‑mini, while also ranking highly on other benchmarks such as LoCoMo and LongMemEval‑S.

By Ahmad Yehia, Aly O. Abdelkareem, Islam Ahmed, Hesham Omran, Khaled Alashmouny, Christian Claudel, Abduallah Mohamed
arXiv AI
Oct 2

Distilling LLM Reasoning into Graph of Concept Predictors

The paper introduces Graph of Concept Predictors (GCP), a reasoning-aware active distillation framework that captures a large language model’s intermediate reasoning as a directed acyclic graph of concepts and mirrors this structure in a smaller student model. GCP improves sample efficiency by using a graph-aware acquisition strategy that weighs concept uncertainty, gradient diversity, and node centrality, and enhances training stability through targeted sub‑module retraining that updates only the most influential concept predictors. Experiments on eight NLP classification benchmarks show that GCP achieves better performance under limited annotation budgets while providing more interpretable and controllable training dynamics.

By Ziyang Yu, Liang Zhao