arXiv Machine Learning July 7, 2026 By Giorgio Racca, Michal Valko, Amartya Sanyal

Language Generation with Replay: A Learning-Theoretic View of Model Collapse

Read the original on arXiv Machine Learning →

arXiv:2603. 11784v2 Announce Type: replace Abstract: As scaling laws push the training of frontier large language models (LLMs) toward ever-growing data requirements, training pipelines are approaching a regime where much of the publicly available online text may be consumed.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

llms

arXiv AI

Jul 1

Learning by Surprise: Adaptive Mitigation of Model Collapse in Large Language Models

arXiv:2410. 12341v4 Announce Type: replace-cross Abstract: As AI-generated content increasingly populates the web, generative AI models are at growing risk of being trained on their own outputs, a process known as AI autophagy.

By Daniele Gambetta, Gizem Gezici, Fosca Giannotti, Dino Pedreschi, Alistair Knott, Luca Pappalardo

llms fine-tuning

arXiv Machine Learning

Jun 30

Generating in the Limit with Infinitely Many Hallucinations

arXiv:2606. 28354v1 Announce Type: cross Abstract: The classic paradigm of language identification in the limit models learning as a game between an adversary, who reveals strings from an unknown target language, and a learner tasked with identifying that language.

By Irene Strauss, Alexandra Butoi, Ryan Cotterell

llms safety

arXiv Machine Learning

Jul 20

Rethinking Transfer in Continual Learning: A Replay-Based Realisation

arXiv:2607. 15587v1 Announce Type: new Abstract: Continual learning studies how deployed language models can continually acquire new tasks without expensive retraining from scratch.

By Yang Meng, Zhenya Liu, Zhuokai Zhao, Yuxin Chen

llms efficiency

arXiv Machine Learning

4d ago

UnlearningSoup: Is Repeated Tuning Necessary for Large Language Model Unlearning?

arXiv:2609.37076v1 Announce Type: new Abstract: Large language models trained on vast corpora inherently risk memorizing harmful content that may later re-emerge in their outputs. To mitigate this is...

By Puning Yang, Qizhou Wang, Junchi Yu, Bo Han, Xiuying Chen

llms

arXiv Machine Learning

Sep 11

A Fragility Spectrum for Recursive Language-Model Training

The paper investigates how recursive contamination—retraining language models on their own generated text—affects output diversity across 13 publicly released checkpoints. Using a fixed contamination protocol over five generations, the authors find a wide spread in 4‑gram diversity (0.187 to 0.940), indicating that some models collapse into repetitive fragments while others remain largely unaffected. The study shows that a model’s susceptibility to collapse is an intrinsic property of the checkpoint, not predicted by parameter scale or static indicators, and that simple interventions such as tightening top‑p sampling can significantly slow or halt collapse.

By Yangze Liu, Zhongyi Han

arXiv AI

4d ago

Break Step: Recursive Training Resonates with Replayed Sampling Noise

arXiv:2609.11149v3 Announce Type: replace-cross Abstract: How fast does a language model degrade when trained on its own outputs? Theory traces it to gradually accumulating errors, while experiments...

By Yangze Liu, Zhongyi Han

llms fine-tuning

One email a morning, machine-written

Related stories

Learning by Surprise: Adaptive Mitigation of Model Collapse in Large Language Models

Generating in the Limit with Infinitely Many Hallucinations

Rethinking Transfer in Continual Learning: A Replay-Based Realisation

UnlearningSoup: Is Repeated Tuning Necessary for Large Language Model Unlearning?

A Fragility Spectrum for Recursive Language-Model Training

Break Step: Recursive Training Resonates with Replayed Sampling Noise