arXiv:2608. 16390v1 Announce Type: cross Abstract: PDF corpora advertise their size in tokens but compute every rate they publish (coverage, OCR routing, re-fetch recovery, language mix) per document, and none decomposes its token total.
arXiv:2608. 00144v1 Announce Type: new Abstract: Membership inference (MIA) on language models is usually summarised by an aggregate ROC-AUC, but such evaluations are confounded: model-free blind baselines separate members from non-members from surface text alone.
The paper investigates how much a masked‑text reconstruction predictor’s outputs would differ if a second corpus were collected from the same population. Using a smoothed predictor based on token‑pair counts, the authors derive a first‑order formula for expected disagreement between predictors trained on two independent corpora, accounting for document length, random context counts, and token‑offset correlations. Experiments on TinyStories, WikiText‑103, and CNN/DailyMail show that increasing corpus size from 512 to 2,048 chunks of 64 tokens reduces disagreement only modestly (5–40 %), and that forecast accuracy improves with larger reference corpora and a source‑split jackknife correction. whyItMatters:"The study provides a practical method to estimate how much predictor outputs would change with additional data, informing decisions about data collection and model robustness in masked‑text reconstruction tasks."
arXiv:2608. 00144v2 Announce Type: replace Abstract: Membership inference (MIA) on language models is usually summarised by aggregate ROC-AUC, but such evaluations are confounded: model-free blind baselines can separate members from non-members using surface text alone.
The paper audits the widely used ISOT/Kaggle Fake and Real News corpus and finds that extremely high reported accuracies (≈0.98) are largely due to shortcut signals rather than genuine veracity detection. A simple TF‑IDF linear classifier achieves perfect F1 when using only subject metadata, and even after removing metadata, newswire tags, and duplicate documents, the F1 drops only modestly, indicating that editorial style rather than specific tokens drives performance. Under topic‑disjoint and temporal transfer tests, performance collapses, and models transfer poorly to the independent LIAR benchmark, showing that within‑corpus scores reflect source and topic separability, not truth verification. whyItMatters:"The study demonstrates that current high accuracy metrics on this fake‑news dataset are misleading, highlighting the need for more robust evaluation protocols that guard against shortcut learning."
arXiv:2607. 24276v1 Announce Type: cross Abstract: Large language models (LLMs) process text through subword tokenizers rather than directly reading characters or words.
arXiv:2608. 09093v1 Announce Type: cross Abstract: How a document's arrangement is written down, its notation, is a training variable that no dataset card records.
arXiv:2608. 05889v1 Announce Type: cross Abstract: Large language models (LLMs) can leave small stylistic traces in text written with their help.
Large language models (LLMs) process text through subword tokenizers rather than directly reading characters or words. Because these tokenizers are trained predominantly on English-centric corpora, they introduce a systematic and often overlooked disadvantage for many non-English languages.
Wieszcz‑XIX is a 6.75‑billion‑token corpus of Polish texts from 1800‑1918, comprising 3.1 billion words across 294,369 documents, assembled from Wolne Lektury and the Internet Archive. The pipeline filters, deduplicates, audits for post‑1918 leakage, and splits at the document level, yielding a training set with only 0.04–0.38 % post‑1918 residue. From this corpus the authors train decoder‑only language models ranging from 47 M to 349 M parameters, demonstrating that larger models better preserve period spelling and exhibit temporal boundedness compared to modern Polish baselines.
The study evaluates machine‑generated text detectors on scientific literature, finding that many detectors perform near perfectly on certain benchmarks due to superficial formatting cues rather than genuine linguistic differences. A logistic regression using 27 superficial features achieved macro‑F1 scores between 0.833 and 0.997, outperforming other detectors on arXiv subsets. The most powerful cue was the PDF extraction artifact—human texts preserve line breaks while generated texts appear as a single block—allowing classification at 0.996 AUC. By introducing format counterfactuals and normalizing formatting, the authors demonstrate that re‑wrapping machine‑written abstracts can flip 20–27% of detector decisions, reducing the generalization gap and improving out‑of‑domain performance.
The paper evaluates CASCADE, a fully local layered defense for Model Context Protocol (MCP)-based systems, by conducting a component ablation and corpus audit on a fixed 5,000-sample dataset. It demonstrates that the choice of aggregation convention heavily influences reported metrics, that detection performance varies with provenance, and that the released configuration does not fully disclose the operating point. The study also shows that a local review model invoked for a third of requests does not alter classification outcomes, highlighting the importance of reproducibility and transparency in defense evaluations.
Re:CAP is a reference‑free audit loop for retrieval‑augmented generation (RAG) pipelines that probes for missing documents instead of enumerating all relevant ones. It identifies covered topics, generates probing questions, retrieves candidate documents, and uses an LLM judge to keep only those that add new information. On several benchmarks, Re:CAP recovers a significant portion of gold documents that flat BM25 or hybrid retrieval misses, and human evaluation shows most of these documents add new information.