← Back to all news
arXiv Computation and Language By Praveen Kumar Myakala, Ravichandra Namburi, Sowmya Keragodu Jayaramu, Sooraj George Thomas

SynthSentry: Detecting Synthetic Data Contamination in Language Model Training Data

Read the original on arXiv Computation and Language →

The paper introduces SynthSentry, a model‑agnostic method for detecting synthetic data contamination in language‑model training corpora. It computes a distributional divergence score based on lexical diversity collapse, n‑gram tail truncation, and perplexity variance across reference models, requiring no access to the generating model or synthetic labels. Experiments on English corpora contaminated by small open‑weight generators and an instruction‑tuned model show that SynthSentry ranks contamination severity accurately, maintains low false‑positive rates after calibration, and does not degrade downstream fine‑tuning performance at the tested scale.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computation and Language.

Related stories

arXiv AI
Oct 9

Obscuring Data Contamination Through Translation: Evidence from Arabic Corpora

The study investigates how exposing large language models to Arabic translations of English benchmark tasks can hide data contamination from standard English-only detection methods. By testing four instruction-tuned models with increasing Arabic exposure, the authors find that traditional probes like TS-Guessing and Min-K%++ fail to detect contamination, yet English performance still improves. They propose a new Translation-Aware Contamination Detection (TACD) method that uses cross-lingual consistency and choice reordering to reveal contamination effects that are otherwise invisible to English-only diagnostics.

By Chaymaa Abbas, Nour Shammaa, Mariette Awad
llmsbenchmarks
More like this →
arXiv AI
Jun 17

Combating Data Laundering in LLM Training

arXiv:2604. 01904v3 Announce Type: replace-cross Abstract: Post-hoc unauthorized-training data detection for large language models (LLMs) typically assumes a query-with-originals regime: rights holders query a target LLM with raw proprietary data and assess whether the model assigns them stronger memorization-based detection signals, e.

By Muxing Li, Zesheng Ye, Sharon Li, Feng Liu
llmsbenchmarks
More like this →
arXiv Computation and Language
Sep 4

Contamination Inflates Scores but Rarely Reorders Large Language Model Leaderboards

The paper investigates how benchmark contamination—leakage of test items into training data—affects large language model (LLM) leaderboards. By comparing original test items with semantically equivalent paraphrases, the authors measure contamination as a violation of anchor-item invariance and find that it inflates absolute scores but rarely changes model rankings. Across 47 public models and 74 finetuned models on four benchmarks, the rank correlation between standard and paraphrase-controlled leaderboards is 0.997, with only a handful of cases showing differential contamination that could alter rankings.

By Xingyao Xiao (Stanford University), Yihong Cheng (City University of Macau)
llmsbenchmarks
More like this →