The paper introduces Word-level Probability MIA (WPMIA), a black-box membership inference attack that estimates word-level generation probabilities via Monte Carlo sampling and local kernel smoothing, then aggregates them into a sequence-level likelihood estimator. By conditioning on different prefixes, WPMIA amplifies distributional differences between member and non-member texts, outperforming existing black-box baselines on open-source LLMs and achieving an average TPR@5%FPR of 42.0 on proprietary models such as GPT‑5‑Chat, Gemini‑2.5‑Flash, and Claude‑4.5‑Haiku.
arXiv:2606. 17464v1 Announce Type: new Abstract: Membership inference attacks (MIAs) are a canonical way to assess a machine learning model's privacy properties.
arXiv:2608. 00144v2 Announce Type: replace Abstract: Membership inference (MIA) on language models is usually summarised by aggregate ROC-AUC, but such evaluations are confounded: model-free blind baselines can separate members from non-members using surface text alone.
arXiv:2608. 00144v1 Announce Type: new Abstract: Membership inference (MIA) on language models is usually summarised by an aggregate ROC-AUC, but such evaluations are confounded: model-free blind baselines separate members from non-members from surface text alone.
Large Language Models (LLMs) raise growing concerns about privacy leakage and copyright compliance. Membership inference is a key tool for assessing such risks, but existing studies mainly focus on whether specific samples or sample-based data units are used for training.
The study re‑runs the OLMo 2 1B language model with controlled training exposure, swapping selected training chunks with matched documents to assess membership inference. Using a counterfactual run from the same seed, the authors find that a single exposure modestly increases a document’s likelihood but rarely allows reliable identification; the reference‑ratio method achieves an AUC of 0.767 and detects 3.8 % of members at a 1 % false‑positive rate. Even with matched references, document‑level variability limits individual inference, and the authors plan to release checkpoints, log‑probabilities, and evaluation data.
arXiv:2606. 24408v1 Announce Type: new Abstract: Assessing the privacy of large language models (LLMs) presents significant challenges.
The paper introduces Pairwise Likelihood MIA (PL‑MIA), a unified membership inference attack that combines a Gaussian likelihood‑ratio statistic with population calibration and the Cauchy combination test. PL‑MIA generates p‑values from pairwise comparisons between a query point and reference points, then aggregates these continuous signals using the Cauchy test to preserve evidence strength. Experiments show that PL‑MIA surpasses strong baselines, boosting true positive rates by over 25% in low‑false‑positive settings, thereby validating the theoretical advantages of the proposed statistical framework.
arXiv:2606. 10481v1 Announce Type: cross Abstract: Parameter-efficient fine-tuning of large language models (LLMs) can exhibit problematic memorization of individual training examples.
The paper introduces a new privacy vulnerability in diffusion language models (DLMs) called token‑level memorization asymmetry, derived from theoretical analysis of diffusion training dynamics. It proposes Q‑Skew, a quantile‑weighted skewness indicator, to perform membership inference on fine‑tuned DLMs, outperforming existing baselines across multiple datasets and models. Additionally, Q‑Skew can be used to extract personally identifiable information (PII), demonstrating a broader privacy attack surface.
The paper introduces a shift‑aware score transport method for privacy auditing that does not require retraining the target model. It estimates the in‑distribution non‑member score distribution from out‑of‑distribution data, transports this relationship to the target covariate distribution, and applies a conservative correction to account for potential extrapolation errors. Experiments on multiple datasets with distribution shifts show that the method provides more stable privacy estimates while maintaining tighter lower bounds than existing baselines.
arXiv:2606. 07996v1 Announce Type: cross Abstract: Pretraining is fundamental to the development of Large Language Models (LLMs), yet the opacity of pretraining data complicates model analysis and raises ethical, legal, and fairness concerns.
arXiv:2506. 06488v3 Announce Type: replace Abstract: A key tool in developing safe AI models is \emph{data auditing}, i.