arXiv Machine Learning October 3, 2026 By Shanbin Yu, Shaoyang Guo, Haoran Zhao, Danni Yu, Ziming Liu

Forking: Sudden Overfitting Under Replay

Read the original on arXiv Machine Learning →

The paper investigates a phenomenon called forking, a type of generalization failure observed in NanoGPT and similar autoregressive models. Forking manifests as a sharp divergence between training and validation loss at epoch boundaries, especially when models employ over‑encoding n‑gram memory modules and data replay. The authors reproduce the effect in both vanilla NanoGPT and a DeepSeek-style model, attributing it to repeated updates that sharpen seen continuations while suppressing unseen ones, with low‑frequency contexts contributing most to the loss gap.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

arXiv Computation and Language

Oct 8

A Deafening Silence: Catastrophic Forgetting Lives in the Output Embeddings of Tokens the Data Never Speaks

arXiv:2610.09835v1 Announce Type: new Abstract: Continual pre-training and fine-tuning in Large Language Models (LLMs) inevitably induce catastrophic forgetting, typically mitigated by replay using o...

By Jonghyun Han, Younghoon Song, Jongyoul Park

llms rag fine-tuning

arXiv Machine Learning

Jun 16

Mechanistic Analysis of Catastrophic Forgetting in Large Language Models During Continual Fine-tuning

arXiv:2601. 18699v2 Announce Type: replace Abstract: Sequential fine-tuning of Large Language Models (LLMs) adaptation to target tasks often triggers catastrophic forgetting, where the acquisition of novel target skills degrades ancestral capabilities.

By Gustav Olaf Yunus Laitinen-Fredriksson Lundstrom-Imanov

llms fine-tuning benchmarks safety

arXiv AI

Jul 17

Per-Token Fixed-Point Convergence in Depth-Recurrent Transformers

arXiv:2607. 14427v1 Announce Type: new Abstract: A depth-recurrent transformer applies a weight-tied core a variable number of times, and prior work has shown that training with a randomized recursion count yields one checkpoint usable across a range of inference depths.

By Joe Logan

llms

arXiv AI

Aug 20

Demystifying Training-Time Augmentation for Data-Constrained Language Model Pretraining

The paper investigates training-time data augmentation as a regularizer for autoregressive language model pretraining in data‑constrained, compute‑abundant settings. It introduces three orthogonal augmentation categories—token‑level noise, sequence permutations, and target offset prediction—and shows through systematic ablations that each category delays overfitting and reduces validation loss, with random token replacement performing best individually. Combining augmentation categories further lowers the minimum validation loss, demonstrating that such augmentations mitigate data inefficiency in autoregressive pretraining.

By Michael K. Chen, Xikun Zhang, Fan Bai, Zhengding Hu, Zhen Wang

llms

arXiv Machine Learning

Jun 9

Lost in the Non-convex Loss Landscape: How to Fine-tune the Large Time Series Model?

arXiv:2606. 08578v1 Announce Type: new Abstract: Recently, large time series models (LTSMs) have gained increasing attention due to their similarities to large language models, including flexible context length, scalability, and task generality, outperforming advanced task-specific models.

By Xu Zhang, Peang Wang, Wei Wang

llms fine-tuning benchmarks

arXiv AI

Jun 18

From Memorization to Parameter Interference: How Overtraining Experts Harms Model Merging

arXiv:2506. 14126v2 Announce Type: replace-cross Abstract: Modern deep learning is increasingly characterized by the use of open-weight foundation models that can be fine-tuned on specialized datasets.

By Stefan Horoi, Guy Wolf, Eugene Belilovsky, Gintare Karolina Dziugaite

fine-tuning

One email a morning, machine-written

Related stories

A Deafening Silence: Catastrophic Forgetting Lives in the Output Embeddings of Tokens the Data Never Speaks

Mechanistic Analysis of Catastrophic Forgetting in Large Language Models During Continual Fine-tuning

Per-Token Fixed-Point Convergence in Depth-Recurrent Transformers

Demystifying Training-Time Augmentation for Data-Constrained Language Model Pretraining

Lost in the Non-convex Loss Landscape: How to Fine-tune the Large Time Series Model?

From Memorization to Parameter Interference: How Overtraining Experts Harms Model Merging