The study investigates which components of a neural network contribute to rapid generalization (grokking) and how stable that improvement remains during further training. By transferring internal attention and MLP weights along with token embeddings and readout, the authors achieve a 5.46‑percentage‑point boost in early accuracy and a 558‑step reduction in confirmation latency, while also demonstrating that freezing transferred representations largely prevents post‑grokking relapse. The work delineates clear component‑level differences between acceleration and stability, and identifies architectural limits where omitting donor embeddings leads to significant performance loss.
By Zeyu Jia
The paper introduces an error‑diffusion (ED) optimizer that replaces the full‑precision latent copy of binary‑weight language models with a small 2‑8‑bit residual buffer, reducing training state per weight from 97 bits to 3‑9 bits—up to 19× less. ED accumulates unrealized weight updates in the buffer and flips the binary weight when the accumulated value crosses a threshold aligned with the quantization grid, enabling efficient training on large transformers with minimal loss compared to latent‑weight baselines. The method scales to models with up to 19 M parameters on enwik8 and 70 M on wikitext‑103, maintaining performance within a few percent of reference while allowing on‑device adaptation within a 275 MB memory budget.
By Pierre-Olivier Bonin
arXiv:2608. 02829v1 Announce Type: new Abstract: Model families train every size from scratch.
By Ravi Satya Durga Prasad Yenugula
arXiv:2608. 11233v1 Announce Type: cross Abstract: A dense, pretrained language model can be retrofitted with recurrent depth and learn an iterative latent transition that persists after outcome-only annealing.
By Mark Shapiro
The paper investigates converting a large pretrained transformer (1.4 B parameters) into a smaller sibling (410 M) by studying representation alignment and parameter projection. It finds that dense weight projection destroys structure, and that a low‑budget, structure‑aware compensation—separating least‑squares function alignment from variance‑preserving rescaling—yields significant gains on token‑efficient training, outperforming subcloning and standard distillation pipelines at matched budgets.
By Ravi Satya Durga Prasad Yenugula
arXiv:2606. 00382v1 Announce Type: new Abstract: Sequential fine-tuning of large language models forces a choice: let the shared substrate keep learning and accept catastrophic forgetting, or freeze it after task one and foreclose cross-task refinement.
By Kiran Nayudu, Aswini Nutakki, Sai Vinay Naidu, Ashwin Shanmugasundaram
NeMo-DCR is a delta‑compressed refit method that sends only the changes in a trillion‑parameter policy model, achieving bit‑exact synchronization across AWS regions. By using fixed affine mappings, XOR masks, and a relay‑tree object storage, it avoids full checkpoint transfers and cross‑cluster collectives, reducing refit times from 87.5 min to 150 s for a 1 T model at 3% change rate. The approach scales to 30 B–1 T models, delivering 12–40× speedups over traditional full‑checkpoint refits.
By Songlin Jiang, Zhiyu Li, Terry Kong, Yu Yao, Youngeun Kwon, Bernard Nguyen, Ashwath Aithal, Mario Di Francesco
TinyCeNN-LM proposes a quality‑gated post‑training conversion framework that replaces attention in pretrained language models with CeNN‑inspired cellular‑recurrent layers. The method includes bounded local processing, compact recurrent memory, routing, fusion, and an accept‑or‑rollback validation step, with three specific implementations studied. Experiments on SmolLM2‑135M and Qwen3.5‑0.8B show that the conversion can accept certain layers while rejecting others based on representation fidelity and NLL thresholds, achieving minimal perplexity changes and modest downstream accuracy retention.
By Kabeh Mohsenzadegan, Vahid Tavakkoli, Kyandoghere Kyamakya
arXiv:2608.20873v1 Announce Type: new
Abstract: Every way of teaching a deployed language model something new -- full fine-tuning, adapter merging, model editing -- replaces the released checkpoint,...
By Zifeng Liu, Zhiyong Du, Yaxin Lu, Yiming Mao, Zhenhe Wang, Wenqi Shi, Zhengkun Jing
arXiv:2607. 15456v1 Announce Type: new Abstract: Looped, weight-tied Transformers reduce parameters by reusing a block, but decoding still stores a separate K/V cache for every recurrence step.
By James O' Neill, Fergal Reid
Muir is a new linear‑attention architecture that consolidates gating mechanisms into a low‑rank projection and uses a matched‑filter readout to re‑weight state rows. It improves over GDN‑2 and Transformers across a wide range of model sizes, lowering perplexity, boosting commonsense accuracy, and enhancing extractive recall while maintaining comparable throughput. The design also includes a numerically stable parallel update for efficient training on large chunk sizes.
By Tushaar Gangavarapu, Gil Tabak, Clemens JS Schaefer
arXiv:2610.16301v1 Announce Type: new
Abstract: Self-attention assigns two functions to one mechanism-where to read and how to mix-at the price of O(L^2) training, an O(T) decode state, and position...
By Helong Hu (Zhanjiang Campus, Guangdong Ocean University, Zhanjiang, Guangdong, China), Hongdan Pan (Maoming Administrative Service Center, Maoming, Guangdong, China), Shuiqing Hu (China Pacific Insurance)