Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

24,507 stories · RSS feed

arXiv Machine Learning

Jun 2

Random Erasing vs. Model Inversion: A Promising Defense or a False Hope?

arXiv:2409. 01062v4 Announce Type: replace Abstract: Model Inversion (MI) attacks pose a significant privacy threat by reconstructing private training data from machine learning models.

By Viet-Hung Tran, Ngoc-Bao Nguyen, Son T. Mai, Hans Vandierendonck, Ira Assent, Alex Kot, Ngai-Man Cheung

benchmarks safety

arXiv AI

Jun 2

What Do LLMs Know About Alzheimer's Disease? Multi-loss Fine-Tuning and Probing for AD Detection

arXiv:2602. 11177v2 Announce Type: replace-cross Abstract: Reliable early detection of Alzheimer's disease (AD) is challenging, particularly due to the limited availability of labeled data.

By Lei Jiang, Yue Zhou, Natalie Parde

llms fine-tuning benchmarks

arXiv AI

Jun 2

BenGER: Benchmarking LLM Systems on Subsumption-Based Legal Reasoning in German Law

arXiv:2605. 28183v2 Announce Type: replace-cross Abstract: We introduce the BenGER (Benchmark for German Law) dataset for evaluating LLM systems on subsumption-based legal reasoning in German law.

By Sebastian Nagl, Ann-Kristin Mayrhofer, Martin Heidebach, Aleyna Ko\c{c}ak, Anne Zettelmeier, Elly Breu, Angelina Greiner, Sofija Milijas, Matthias Grabmair

llms benchmarks

arXiv AI

Jun 2

Genie 4D: Semantic-Prior-Guided 4D Dynamic Scene Reconstruction

arXiv:2604. 09877v2 Announce Type: replace-cross Abstract: At the intersection of computer vision and robotic perception, 4D reconstruction of dynamic scenes connects low-level geometric sensing with high-level semantic understanding.

By Yiru Yang, Zhuojie Wu, Nishant Kumar Singh, Max Schulthess

rag diffusion computer-vision robotics benchmarks

arXiv AI

Jun 2

ExpertGen: Scalable Sim-to-Real Expert Policy Learning from Imperfect Behavior Priors

arXiv:2603. 15956v3 Announce Type: replace-cross Abstract: Learning generalizable and robust behavior cloning policies requires large volumes of high-quality robotics data.

By Zifan Xu, Ran Gong, Maria Vittoria Minniti, Kausik Sivakumar, Ahmet Salih Gundogdu, Eric Rosen, Riedana Yan, Tushar Kusnur, Zixing Wang, Di Deng, Peter Stone, Xiaohan Zhang, Karl Schmeckpeper

llms diffusion reinforcement-learning robotics benchmarks

arXiv AI

Jun 2

AgentRedBench: Dynamic Redteaming and Integration-Aware Defense for LLM Agents over SaaS Integrations

arXiv:2606. 02240v1 Announce Type: cross Abstract: Indirect prompt injection in tool-use agents is a concrete production threat: LLM agents read from integrations (third-party services such as Gmail, Salesforce, or Jira accessed through tool calls) whose response content the user neither writes nor controls.

By Hiskias Dingeto, Will Leeney

llms agents multimodal benchmarks safety

arXiv AI

Jun 2

Brain-Atlas-Guided Generative Counterfactual Attention for Explainable Cognitive Decline Diagnosis Using Multimodal Connectomes

arXiv:2606. 01237v1 Announce Type: new Abstract: Mild cognitive impairment (MCI) and subjective cognitive decline (SCD) are closely associated with the early Alzheimer's disease continuum, where accurate and explainable diagnosis is important for early risk assessment and intervention.

By Xiongri Shen, Jiaqi Wang, Zhenxi Song, Yi Zhong, Leilei Zhao, Xin He, Baiying Lei, Zhiguo Zhang

llms multimodal benchmarks safety

arXiv Machine Learning

Jun 2

Beyond the Simplex: Balanced Prototype Geometry for Scorer-Agnostic Open-Set Recognition

arXiv:2606. 01883v1 Announce Type: new Abstract: Open-set recognition (OSR) requires a classifier to reject inputs from unseen classes which is essential in safety-critical settings such as medical imaging.

By Mayank Sharma, Rohit Kumar Mourya

rag benchmarks safety

arXiv Machine Learning

Jun 2

Identifiable Markov Switching Models with Instantaneous Effects and Exponential Families

arXiv:2606. 02231v1 Announce Type: cross Abstract: Temporal systems often exhibit non-stationary behaviour, such as seasonal climate variation or glucose fluctuations in patients with type-1 diabetes.

By Roel Hulsman, Carles Balsells-Rodas, Sara Magliacane

benchmarks

arXiv Machine Learning

Jun 2

A combination of noise and bilateral filters achieve supralinear and scalable adversarial robustness in CNNs

arXiv:2606. 02267v1 Announce Type: new Abstract: The vulnerability of deep neural networks to adversarial examples poses a significant challenge for real-world deployment.

By Nicolas Stalder, Benjamin F. Grewe, Matteo Saponati, Pau Vilimelis Aceituno

benchmarks safety

arXiv Machine Learning

Jun 2

Dimensionality Reduction for Robust Federated Learning: A Theoretical Analysis and Convergence Guarantee

arXiv:2605. 28335v2 Announce Type: replace Abstract: Federated Learning (FL) enables multiple clients to collaboratively train models without sharing raw data, but it is highly vulnerable to Byzantine attacks.

By Shiyuan Zuo, Jiashuo Li, Rongfei Fan, Han Hu, Jie Xu

rag benchmarks

arXiv Machine Learning

Jun 2

Learning-based Directed Graph Abstraction of Combinatorial Spaces for Order-Preserving Search in Mixed-Combinatorial Nonlinear Optimization

arXiv:2606. 01425v1 Announce Type: new Abstract: Mixed-combinatorial nonlinear programming (MCNLP) problems arise in many engineering design and planning applications, e.

By Gishnu Madhu, Feng Liu, Souma Chowdhury

robotics benchmarks

arXiv Machine Learning

Jun 2

Interventional Processes for Causal Uncertainty Quantification

arXiv:2410. 14483v3 Announce Type: replace-cross Abstract: Reliable uncertainty quantification for causal effects is crucial in high-stakes applications, but remains challenging when the target is an entire function rather than a scalar estimand.

By Hugh Dance, Peter Orbanz, Arthur Gretton

benchmarks

arXiv AI

Jun 2

Lying Is Just a Phase: The Hidden Alignment Transition in Language Model Scaling

arXiv:2605. 18838v3 Announce Type: replace-cross Abstract: Scaling laws predict loss from compute but not how capabilities interact.

By Adil Amin

llms efficiency benchmarks safety

arXiv Machine Learning

Jun 2

FinTSB: A Comprehensive and Practical Benchmark for Financial Time Series Forecasting

arXiv:2502. 18834v3 Announce Type: replace-cross Abstract: Financial time series (FinTS) record the behavior of human-brain-augmented decision-making, capturing valuable historical information that can be leveraged for profitable investment strategies.

By Yifan Hu, Yuante Li, Peiyuan Liu, Yuxia Zhu, Naiqi Li, Tao Dai, Shu-tao Xia, Dawei Cheng, Changjun Jiang

benchmarks data-engineering

arXiv AI

Jun 2

MCP-Persona: Benchmarking LLM Agents on Real-World Personal Applications via Environment Simulation

arXiv:2606. 02470v1 Announce Type: new Abstract: The Model Context Protocol (MCP) has emerged as a transformative standard for connecting large language models (LLMs) with external data sources and tools, and has been rapidly adopted across personal applications and development platforms.

By Wenhao Wang, Peizhi Niu, Gongyi Zou, Xiyuan Yang, Jingxing Wang, Haoting Shi, Yaxin Du, Jingyi Chai, Xianghe Pang, Shuo Tang, Yanfeng Wang, Siheng Chen

llms agents benchmarks

arXiv Machine Learning

Jun 2

Embedding-Space Diffusion for Zero-Shot Environmental Sound Classification

arXiv:2412. 03771v3 Announce Type: replace-cross Abstract: Zero-shot learning enables models to generalise to unseen classes by leveraging semantic information, bridging the gap between training and testing sets with non-overlapping classes.

By Ysobel Sims, Alexandre Mendes, Stephan Chalup

rag diffusion computer-vision benchmarks safety

arXiv AI

Jun 2

Connecting the Dots: Benchmarking Reflective Memory in Long-Horizon Dialogue

arXiv:2606. 01223v1 Announce Type: cross Abstract: Despite substantial progress in long-context modeling, existing benchmarks remain confined to factual memory for explicit recall, failing to measure the reflective memory required to synthesize fragmented, multimodal cues into high-level interpretations.

By Jingjie Lin, Bingbing Wang, Zihan Wang, Zhengda Jin, Weiming Qiao, Jing Li, Ruifeng Xu

multimodal benchmarks safety

arXiv AI

Jun 2

SPADE-Bench: Evaluating Spontaneous Strategic Deception in Agents via Plan-Action Divergence

arXiv:2606. 02380v1 Announce Type: cross Abstract: As LLM-based agents expand their operational scope, reliability becomes a prerequisite for real-world deployment.

By Yuyan Bu, Haowei Li, Qirui Zheng, Bowen Dong, Kaiyue Yang, Jiaming Ji, Yingshui Tan, Wenxin Li, Yaodong Yang, Juntao Dai

llms agents benchmarks safety

arXiv AI

Jun 2

Product-Aware Deep Autoencoders for Robust Process Monitoring in Multi-Product Cyber-Physical Systems

arXiv:2606. 00052v1 Announce Type: new Abstract: As Industry 4.

By MD Shafikul Islam, Jordan Carden

benchmarks safety

Get the daily briefing