arXiv AI
Benefit and Cost of Random-Token Replacement under Repeated Data: A Known-Entropy Decomposition and Evidence from Ten Real Corpora
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv Machine Learning
Jun 25
Internal Data Repetition Destroys Language Models
arXiv:2606. 24998v1 Announce Type: new Abstract: Language models are running out of high-quality training data, and even aggressively deduplicated corpora retain some amount of repetition.
arXiv Computation and Language
Sep 2
Removable and Irreducible: A Token-Cost Ledger for the Multilingual Tokenization Tax
arXiv:2609.00378v1 Announce Type: new Abstract: Large language models pay a well-documented tax on non-English text: the same content costs several times more tokens, and because attention is quadrat...
arXiv Computation and Language
Oct 8
A Deafening Silence: Catastrophic Forgetting Lives in the Output Embeddings of Tokens the Data Never Speaks
arXiv:2610.09835v1 Announce Type: new Abstract: Continual pre-training and fine-tuning in Large Language Models (LLMs) inevitably induce catastrophic forgetting, typically mitigated by replay using o...
arXiv Machine Learning
Jun 5
SHRED: Retain-Set-Free Unlearning via Self-Distillation with Logit Demotion
arXiv:2605. 07482v2 Announce Type: replace Abstract: Machine unlearning for large language models (LLMs) aims to selectively remove memorized content such as private data, copyrighted text, or hazardous knowledge, without costly full retraining.
arXiv Machine Learning
Jul 15
Extractable Memorization From First Principles
arXiv:2607. 12649v1 Announce Type: new Abstract: Recent work on extractable memorization in LLMs suffers from two contrasting validity problems.
arXiv Machine Learning
Oct 12
Forecasting Corpus-Sampling Disagreement in Masked Text Reconstruction
The paper investigates how much a masked‑text reconstruction predictor’s outputs would differ if a second corpus were collected from the same population. Using a smoothed predictor based on token‑pair counts, the authors derive a first‑order formula for expected disagreement between predictors trained on two independent corpora, accounting for document length, random context counts, and token‑offset correlations. Experiments on TinyStories, WikiText‑103, and CNN/DailyMail show that increasing corpus size from 512 to 2,048 chunks of 64 tokens reduces disagreement only modestly (5–40 %), and that forecast accuracy improves with larger reference corpora and a source‑split jackknife correction. whyItMatters:"The study provides a practical method to estimate how much predictor outputs would change with additional data, informing decisions about data collection and model robustness in masked‑text reconstruction tasks."