arXiv:2608. 00144v1 Announce Type: new Abstract: Membership inference (MIA) on language models is usually summarised by an aggregate ROC-AUC, but such evaluations are confounded: model-free blind baselines separate members from non-members from surface text alone.
By Victor Maricato
The paper investigates how much a masked‑text reconstruction predictor’s outputs would differ if a second corpus were collected from the same population. Using a smoothed predictor based on token‑pair counts, the authors derive a first‑order formula for expected disagreement between predictors trained on two independent corpora, accounting for document length, random context counts, and token‑offset correlations. Experiments on TinyStories, WikiText‑103, and CNN/DailyMail show that increasing corpus size from 512 to 2,048 chunks of 64 tokens reduces disagreement only modestly (5–40 %), and that forecast accuracy improves with larger reference corpora and a source‑split jackknife correction.
whyItMatters:"The study provides a practical method to estimate how much predictor outputs would change with additional data, informing decisions about data collection and model robustness in masked‑text reconstruction tasks."
By Hare Krishna
arXiv:2608. 00144v2 Announce Type: replace Abstract: Membership inference (MIA) on language models is usually summarised by aggregate ROC-AUC, but such evaluations are confounded: model-free blind baselines can separate members from non-members using surface text alone.
By Victor Maricato
The paper audits the widely used ISOT/Kaggle Fake and Real News corpus and finds that extremely high reported accuracies (≈0.98) are largely due to shortcut signals rather than genuine veracity detection. A simple TF‑IDF linear classifier achieves perfect F1 when using only subject metadata, and even after removing metadata, newswire tags, and duplicate documents, the F1 drops only modestly, indicating that editorial style rather than specific tokens drives performance. Under topic‑disjoint and temporal transfer tests, performance collapses, and models transfer poorly to the independent LIAR benchmark, showing that within‑corpus scores reflect source and topic separability, not truth verification.
whyItMatters:"The study demonstrates that current high accuracy metrics on this fake‑news dataset are misleading, highlighting the need for more robust evaluation protocols that guard against shortcut learning."
By Yuvraj Verma
arXiv:2607. 24276v1 Announce Type: cross Abstract: Large language models (LLMs) process text through subword tokenizers rather than directly reading characters or words.
By Priyansh Srivastava
arXiv:2608. 09093v1 Announce Type: cross Abstract: How a document's arrangement is written down, its notation, is a training variable that no dataset card records.
By E. M. Freeburg