arXiv Machine Learning

Benchmarking Classical and Transformer-Based Models for Document Sensitivity Classification

The paper introduces Strategic 16K, a 16,000‑document corpus of diplomatic cables from WikiLeaks’ Public Library of US Diplomacy, designed to eliminate label leakage. It benchmarks six models—both classical machine learning and transformer-based—on this leakage‑controlled dataset, finding BERT and ELECTRA top performers while TF‑IDF with Logistic Regression offers strong accuracy at lower cost. This work provides the first fully reproducible sensitivity‑classification benchmark built under explicit leakage‑control conditions.

arXiv Machine Learning

Sep 23

What Does 99% Accuracy Measure? A Reproducible Audit of Shortcut Learning in a Widely Used Fake News Corpus

The paper audits the widely used ISOT/Kaggle Fake and Real News corpus and finds that extremely high reported accuracies (≈0.98) are largely due to shortcut signals rather than genuine veracity detection. A simple TF‑IDF linear classifier achieves perfect F1 when using only subject metadata, and even after removing metadata, newswire tags, and duplicate documents, the F1 drops only modestly, indicating that editorial style rather than specific tokens drives performance. Under topic‑disjoint and temporal transfer tests, performance collapses, and models transfer poorly to the independent LIAR benchmark, showing that within‑corpus scores reflect source and topic separability, not truth verification. whyItMatters:"The study demonstrates that current high accuracy metrics on this fake‑news dataset are misleading, highlighting the need for more robust evaluation protocols that guard against shortcut learning."

By Yuvraj Verma

fine-tuning benchmarks

arXiv Computation and Language

Sep 15

PARSE: Provenance-Aware Retrieval Sanitization for Professional Domain LLM Agents

arXiv:2606.17467v3 Announce Type: replace-cross Abstract: Prompt injection defenses evaluated on synthetic benchmarks do not generalize to real enterprise documents, which are longer, denser, and int...

By Aaditya Pai

llms agents benchmarks

arXiv Computation and Language

Sep 22

From Articles to Publishers: Aggregating Language Model Predictions for News Source Reliability Inference

arXiv:2609.24219v1 Announce Type: new Abstract: Traditionally, the reliability of news publishers is assessed by expert organisations that evaluate editorial practices, transparency and factual stand...

By John Bianchi, Manuel Pratelli, Fabio Pinelli, Marinella Petrocchi

llms

arXiv Machine Learning

Aug 12

Leak It: Per-Document Extraction Beyond Aggregate Membership Inference

arXiv:2608. 00144v2 Announce Type: replace Abstract: Membership inference (MIA) on language models is usually summarised by aggregate ROC-AUC, but such evaluations are confounded: model-free blind baselines can separate members from non-members using surface text alone.

By Victor Maricato

llms safety

arXiv Machine Learning

Aug 4

Leak It: A Probabilistic Approach to Training-Data Extraction from Black-Box Language Models

arXiv:2608. 00144v1 Announce Type: new Abstract: Membership inference (MIA) on language models is usually summarised by an aggregate ROC-AUC, but such evaluations are confounded: model-free blind baselines separate members from non-members from surface text alone.

By Victor Maricato

llms safety

arXiv AI

Aug 3

TextCloak: Thwarting Unauthorized LLM Exploitation via RL-Driven Unlearnable Text

arXiv:2607. 28862v1 Announce Type: cross Abstract: The rapid development of Large Language Models (LLMs) has led to significant advances across a wide range of language tasks, while simultaneously raising growing concerns about unauthorized data exploitation and privacy leakage.

By Chengshuai Zhao, Pingchuan Ma, Dawei Li, Bohan Jiang, Zhiyuan Yu, Zhen Tan, Huan Liu

llms nlp reinforcement-learning fine-tuning benchmarks safety

arXiv Computation and Language

Sep 3

LeakageBench: Document-Level Leakage Risk for Redacting Personally Identifiable Information in Document Images

LeakageBench is a new benchmark comprising 500 document images with 11,954 GDPR‑aligned PII annotations, designed to evaluate document‑level redaction risk. It measures how well OCR pipelines, OCR‑dependent detectors, and OCR‑free vision‑language models can localize and remove sensitive information, using entity‑level F1, group‑wise leakage, and document‑level leakage metrics. The study shows that while advanced models improve localization, most pages still exhibit critical leakage, highlighting the need for higher‑recall, spatially grounded redaction methods.

By Vishnu Prasad Vijaya Kumar, Santhosh Venkatesh, Ivan P. Yamshchikov

llms computer-vision multimodal benchmarks

arXiv AI

Sep 1

Measuring the Depth of LLM Unlearning via Activation Patching

arXiv:2605.24614v2 Announce Type: replace-cross Abstract: Large language model (LLM) unlearning has emerged as a crucial post-hoc mechanism for privacy protection and AI safety, yet auditing whether...

By Jaeung Lee, Dohyun Kim, Jaemin Jo

llms benchmarks safety

arXiv AI

Aug 11

Build it, Break it, Repeat: Benchmarking and improving LLM-manipulated disinformation detection in social media posts

arXiv:2608. 09510v1 Announce Type: cross Abstract: Detecting machine-generated disinformation on social media is increasingly difficult as large language models (LLMs) make it easier to generate and rewrite misleading content at scale.

By Kevin Thomas, Milosz Kasprzyk, Reuel C Igbokwe Onuigbo, Elliott Pert, Cameron Tovey, Jo\~ao A. Leite, Olesya Razuvayevskaya, Carolina Scarton

llms fine-tuning benchmarks safety

arXiv Machine Learning

Jun 2

Unlearning Isn't Invisible: Detecting Unlearning Traces in LLMs from Model Outputs

arXiv:2506. 14003v5 Announce Type: replace Abstract: Machine unlearning (MU) for large language models (LLMs), commonly referred to as LLM unlearning, seeks to remove specific undesirable data or knowledge from a trained model, while maintaining its performance on standard tasks.

By Yiwei Chen, Soumyadeep Pal, Yimeng Zhang, Qing Qu, Sijia Liu

llms safety

arXiv AI

Jun 16

CHILLGuard: Towards Fine-Grained Chinese LLM Safety Guardrail with Scalable Data Construction and Model-aware Preference Alignment

arXiv:2606. 15396v1 Announce Type: cross Abstract: Malicious content generated from large language models (LLMs) could pose severe safety risks and ethical concerns.

By Wenbo Yu, Bohua Wang, Hao Fang, Kuofeng Gao, Jingru Zeng, Xiaochen Yang, Tianyi Zhang, Xiaoxiao Ma, Jiawei Kong, Hao Wu, Bin Chen, Shu-Tao Xia, Min Zhang

llms rag benchmarks safety

Hugging Face Trending Papers

Sep 2

LeakageBench: Document-Level Leakage Risk for Redacting Personally Identifiable Information in Document Images

LeakageBench is a new benchmark consisting of 500 document images with 11,954 GDPR‑aligned PII annotations, designed to evaluate document‑level redaction risk. Unlike existing text‑centric PII benchmarks, it measures whether a page remains unsafe if any identifier is missed, using entity‑level F1, group‑wise leakage, and document‑level leakage metrics. Experiments show that even advanced OCR pipelines and vision‑language models improve localization but still leave a high proportion of pages unsafe for release.

llms computer-vision multimodal benchmarks

Related stories