Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

15,829 stories · RSS feed

arXiv Machine Learning

Jul 23

Classical Hardware Acceleration of Quantum Autoencoders for Real-Time Anomaly Detection in Collider Experiments

arXiv:2607. 20302v1 Announce Type: new Abstract: Quantum machine learning (QML) algorithms in high energy physics (HEP) can efficiently represent and leverage long-range, high-order correlations in high-dimensional collider data, potentially with fewer parameters and favorable scaling relative to classical models.

By Ivan Ge, Sagar Addepalli, Abhilasha Dave, Julia Gonski

benchmarks

arXiv AI

Jul 23

NeuroSymActive: Differentiable Neural-Symbolic Reasoning with Active Exploration for Knowledge Graph Question Answering

arXiv:2602. 15353v3 Announce Type: replace-cross Abstract: Large pretrained language models and neural reasoning systems have advanced many natural language tasks, yet they remain challenged by knowledge-intensive queries that require precise, structured multi-hop inference.

By Rong Fu, Yang Li, Zeyu Zhang, Jiekai Wu, Yaohua Liu, Shuaishuai Cao, Yangchen Zeng, Yuhang Zhang, Xiaojing Du, Simon Fong

llms rag nlp benchmarks

arXiv Machine Learning

Jul 23

Membership Inference Attacks for Unseen Classes

arXiv:2506. 06488v3 Announce Type: replace Abstract: A key tool in developing safe AI models is \emph{data auditing}, i.

By Pratiksha Thaker, Neil Kale, Zhiwei Steven Wu, Virginia Smith

benchmarks safety

arXiv AI

Jul 23

NEXUS: Structured Runtime Safety for Tool-Using LLM Agents

arXiv:2607. 19356v1 Announce Type: new Abstract: Tool-using LLM agents increasingly execute high-impact actions, making runtime safety monitoring essential.

By Elias Hossain, Md Mehedi Hasan Nipu, Tasfia Nuzhat Ornee, Rajib Rana, Niloofar Yousefi

llms agents multimodal benchmarks safety

arXiv AI

Jul 23

Geometry-Guided Constraint Learning for LLM Safety Classification

arXiv:2607. 19366v1 Announce Type: new Abstract: Safety as Polytope (SaP) learns linear half-space constraints in LLM hidden space but requires per-category tuning of the constraint count K.

By Fumiaki Uehara, Koo Imai, Masato Tsutsumi, Keigo Kansa, Sora Usui, Yuki Kobiyama

llms benchmarks safety

arXiv AI

Jul 23

Rewarding Better Thinking for LLM Preference Alignment

arXiv:2607. 19824v1 Announce Type: new Abstract: LLM preference alignment aims to optimize models toward human preferences across diverse user instructions.

By Xubo Liu, Wenya Guo, Ruxue Yan, Xinying Qian, Ying Zhang

llms reinforcement-learning benchmarks safety

arXiv Machine Learning

Jul 23

STN-TGAT: Top-K Portfolio Construction via Prior-Guided Graph Attention with Learnable Soft-Threshold Sparsification

arXiv:2607. 19385v1 Announce Type: new Abstract: This paper tackles the problem of stock ranking and portfolio construction under realistic investment settings by jointly modeling temporal dynamics and cross-sectional dependencies.

By Haoran Guo, Yutong Lu, Li Zhang

llms benchmarks

arXiv AI

Jul 23

Closing the Lab-to-Store Gap: A Data-Efficient Post-Training and Experience-Driven Learning VLA Framework for Retail Humanoids

arXiv:2607. 20345v1 Announce Type: cross Abstract: Closing the gap between benchmark performance and reliable real-world operation remains a central challenge for Vision-Language-Action (VLA) humanoid robots, which must handle execution errors, distribution shifts, and environmental variability.

By Roger Sala Sis\'o, Tiago Silv\'erio, Jakob Sand, Tran Nguyen Le

robotics fine-tuning multimodal benchmarks safety

arXiv AI

Jul 23

PhenSPINE: A Standardized Benchmark for Spine Pathology Diagnosis

arXiv:2607. 19696v1 Announce Type: cross Abstract: The accurate diagnosis of spinal pathologies depends heavily on radiological interpretation, yet automated systems are hindered by the lack of diverse, high-quality benchmarks.

By Duong Ngoc Vu, Hai Son Nguyen, Trong-Nghia Nguyen, Bien Tran Van, Trang Mai Xuan, Huan Vu, Thien Van Luong

benchmarks

arXiv Machine Learning

Jul 23

Trend strength predicts when generative foundation models win: a power-controlled benchmark, a mechanism, and an actionable selection rule

arXiv:2607. 19383v1 Announce Type: cross Abstract: Pretrained generative foundation models cast forecasting as conditional generation from a learned predictive distribution and forecast unseen series zero-shot.

By Ahmed Cherif

benchmarks

arXiv Machine Learning

Jul 23

Harnessing Disagreement: Detecting Correlated Agreement Blindness in Multi-Agent Triage

arXiv:2607. 19899v1 Announce Type: cross Abstract: Disagreement-triggered escalation can create a structural blind spot in multi-agent arbitration: as base learners improve, they tend to converge, weakening safety monitoring where correlated failures concentrate.

By Shay Seiya McDonnell, Avantika Singh, Quoc-Viet Pham, Vratislav Havlik, Gregory M. P. O'Hare

agents benchmarks safety

arXiv Machine Learning

Jul 23

SEED: Towards More Accurate Semantic Evaluation for Visual Brain Decoding

arXiv:2503. 06437v3 Announce Type: replace-cross Abstract: We present SEED (Semantic Evaluation for Visual Brain Decoding), a novel metric for evaluating the semantic decoding performance of visual brain decoding models.

By Juhyeon Park, Peter Yongho Kim, Jiook Cha, Shinjae Yoo, Taesup Moon

benchmarks safety

arXiv AI

Jul 23

Defense Against LLM Backdoors using Critical Neuron Isolation Pruning

arXiv:2607. 19894v1 Announce Type: cross Abstract: Large language models (LLMs) are vulnerable to backdoor attacks, where hidden triggers induce malicious outputs.

By Yuxi Li, Zhibo Zhang, Kailong Wang, Xingshuo Han, Ling Shi, Haoyu Wang

llms fine-tuning efficiency multimodal benchmarks

arXiv Machine Learning

Jul 23

Label-Noise Resistant Learning via Optimal Brain Damage Masking

arXiv:2508. 09697v4 Announce Type: replace Abstract: Noisy labels are inevitable in real-world multimedia applications.

By Xinlei Zhang, Fan Liu, Chuanyi Zhang, Xiaoying Ji, Wenhui Wang, Wei Zhou, Yuhui Zheng

efficiency benchmarks

arXiv AI

Jul 23

JailMeter: An Evidence-Based Evaluation Framework for Jailbreak Attacks on Large Language Models

arXiv:2607. 19424v1 Announce Type: cross Abstract: The assessment of jailbreak attacks against large language models currently suffers from inconsistent evaluation criteria and methods, leading to unreliable estimates of attack success rates.

By Qingjia Huang, Jingyu Zhang, Jianguo Wu, Yakai Li, Weijuan Zhang, Yankai Rong, Junyi Yao, Shengzhi Zhang, Xiaoqi Jia

llms benchmarks safety

arXiv AI

Jul 23

When Does Knowledge Distillation Hurt? Reliability-Aware Distillation for Low-Resource Language Summarization

arXiv:2607. 19956v1 Announce Type: cross Abstract: Knowledge distillation (KD) is a standard approach for compressing sequence-to-sequence models, but its per-sample effects are rarely examined.

By Dipto Sumit, Ankan Kumar Roy Srizon, Sadia Khair Rodela, Atia Haque Asha, Mourchona Afrin, Niloy Farhan, Farig Sadeque

nlp fine-tuning efficiency benchmarks safety

arXiv Machine Learning

Jul 23

JAGG: Jacobian-Aggregated Group Gradient for Efficient GRPO Training of Diffusion Models

arXiv:2607. 17572v2 Announce Type: replace Abstract: Group Relative Policy Optimization (GRPO) is a powerful reinforcement learning algorithm for aligning generative models with human preferences.

By Ruiyi Ding, Jie Li, He Kang, Ziyan Liu, Chengru Song, Yuan chen

llms diffusion reinforcement-learning benchmarks

arXiv Machine Learning

Jul 23

Q-PhotoNAS: Hybrid Quantum Neural Architecture Search Framework on Photonic Devices

arXiv:2605. 22097v2 Announce Type: replace-cross Abstract: Photonic quantum computing is a promising platform for scalable quantum machine learning, but designing effective hybrid architectures remains challenging under hardware and optimization constraints.

By Farah Elnakhal, Alberto Marchisio, Nouhaila Innan, Gabriel Falcao, Muhammad Shafique

computer-vision benchmarks

arXiv Machine Learning

Jul 23

NeuCoReClass AD: Redefining Self-Supervised Time Series Anomaly Detection

arXiv:2508. 00909v2 Announce Type: replace Abstract: Time series anomaly detection plays a critical role in a wide range of real-world applications.

By Aitor S\'anchez-Ferrera, Usue Mori, Borja Calvo, Jose A. Lozano

benchmarks

arXiv Machine Learning

Jul 23

Directional Kernel Mean Difference: A Fast Signed Statistic for Univariate Distribution Comparison

arXiv:2607. 20119v1 Announce Type: cross Abstract: We introduce the Directional Kernel Mean Difference (DKMD), a signed statistic for univariate distribution comparison that preserves the direction of distributional shifts.

By Shijie Zhong, Jiangfeng Fu

rag benchmarks

Get the daily briefing