arXiv AI

The Reliability Gap in Benchmark Auditing: Distribution Shift and Scale as Failure Modes of Contamination Detection

arXiv:2606. 03305v1 Announce Type: new Abstract: Benchmark contamination, where evaluation examples appear in a model's training data, threatens the validity of LLM assessment.

arXiv AI

Jul 28

Beyond Shapley: An Influence-Based Data Auditing Pipeline for LLM Alignment and Evaluation

arXiv:2607. 22766v1 Announce Type: cross Abstract: The alignment of Large Language Models (LLMs) is increasingly bottlenecked by data quality.

By Yunting Song, Matthew Watson, Peter Grabowski, Jun Qin

llms reinforcement-learning benchmarks safety data-engineering

arXiv Machine Learning

Aug 4

Crushing the Evidence: A Dual-Penalty Evasion Framework for Fooling White-Box Explainable AI Auditors

arXiv:2608. 00566v1 Announce Type: new Abstract: Post-hoc model explainers such as LIME, SHAP, and Integrated Gradients are widely deployed to audit models in high-stakes sensitive domains, including finance, healthcare, and social welfare.

By Niraj Kumar, Harsh Kasyap

rag benchmarks safety

arXiv Machine Learning

Jun 11

A prior-free blind detection of information leakage from model predictions

arXiv:2606. 11267v1 Announce Type: new Abstract: Data leakage -- contamination of a model with information unavailable at baseline -- is the dominant reproducibility failure in machine-learning-based science, yet detection tools require training code, external data, or domain expertise.

By Laurence A. Jacobs

arXiv Machine Learning

Jun 16

Auditing Machine Unlearning: A Systematic Research on Whether Models Truly Forget

arXiv:2606. 16110v1 Announce Type: new Abstract: Machine unlearning has been extensively studied in response to growing privacy concerns and regulatory requirements.

By Dayong Ye, Tianqing Zhu, Ruiding Huang, Xinbo Fu, Jiayang Li, Bo Liu, Huan Huo, Wanlei Zhou

llms fine-tuning safety

arXiv AI

Jul 29

Stemma: Induced Decision Regions Reveal LLM Provenance

arXiv:2607. 25880v1 Announce Type: cross Abstract: LLM provenance testing asks whether a suspect LLM belongs to the same lineage as a source.

By Keyu Zhang, Vadim Safronov, Andrew Martin

llms

arXiv Machine Learning

Jul 21

When Benchmarks Lie: Evaluating Malicious Prompt Classifiers Under True Distribution Shift

arXiv:2602. 14161v2 Announce Type: replace Abstract: Detecting prompt injection, jailbreak attacks, and harmful requests is critical for deploying LLM-based agents safely, yet current evaluation practices in this literature overestimate generalization.

By Max Fomin

llms agents benchmarks safety

arXiv Machine Learning

Aug 4

How Benchmarks and Evaluation Protocols Shape Conclusions in Provenance-Based Intrusion Detection

arXiv:2608. 01454v1 Announce Type: cross Abstract: Provenance-based intrusion detection systems (PIDS) frequently report strong performance, but the conclusions drawn from these results can be highly sensitive to benchmarking choices and evaluation protocols.

By Lorenzo Guerra, Thomas Chapuis, Guillaume Duc, Pavlo Mozharovskyi, Van-Tam Nguyen

benchmarks

Hugging Face Trending Papers

Jun 15

Auditing Machine Unlearning: A Systematic Research on Whether Models Truly Forget

Machine unlearning has been extensively studied in response to growing privacy concerns and regulatory requirements. However, auditing whether unlearning algorithms have truly erased the influence of specific data remains an open challenge.

llms fine-tuning safety

arXiv AI

Jun 3

Auditable Climate Risk Intelligence from Fragmented ESG Data: Deterministic Orchestration and Imbalance-Aware Learning for Scope 1-3 Validation

arXiv:2606. 02604v1 Announce Type: cross Abstract: ESG and climate risk data remain fragmented across heterogeneous Scope 1, Scope 2, and Scope 3 reporting environments, while conventional validation pipelines lack provenance aware auditability, hidden drift detection, and reproducibility oriented governance.

By Karan Sehgal, Khawar Naveed Bhatti

agents benchmarks safety

arXiv Machine Learning

Jul 24

DataPrep-Bench: Benchmarking LLMs as Training Data Preparators

arXiv:2607. 20465v1 Announce Type: new Abstract: The quality of training data fundamentally determines the capabilities of large language models (LLMs), yet no unified benchmark exists to measure how well LLMs, agents, and data-centric workflows actually prepare training data end to end.

By Hao Liang, Qifeng Cai, Yibo Lin, Jianzhuo Du, Qifeng Xia, Sizhe Qiu, Linzhuang Sun, Meiyi Qiang, Zhaoyang Han, Xiaochen Ma, Bohan Zeng, Ruichuan An, Conghui He, Wentao Zhang

llms agents fine-tuning benchmarks safety data-engineering

arXiv Machine Learning

Jul 15

Auditing Data Leakage in Whole-Slide Image Multimodal Benchmarks

arXiv:2607. 12278v1 Announce Type: cross Abstract: Recent vision-language models (VLMs) for computational pathology report striking zero-shot performance on whole-slide image (WSI) visual question answering (VQA) benchmarks.

By Wenhao Zhang, Zhongliang Zhou, John Kang, Sheng Li

llms nlp multimodal benchmarks

arXiv AI

Jul 16

Operational Evidence Gaps for LLMs in Fraud Detection and Trust-and-Safety Workflows

arXiv:2607. 13078v1 Announce Type: cross Abstract: LLMs are now proposed for fraud detection, scam investigation, content moderation, and other trust-and-safety workflows.

By Keyur Gabani

llms agents benchmarks safety

Related stories