arXiv Machine Learning

Beyond Embedding Transfer: Architectural Interfaces, Iso-Parameter Scaling, and Preconditioner-Governed Consolidation in Grokking

arXiv Machine Learning
Sep 17

Beyond Embedding Transfer: Component Roles in Grokking Transfer and Stability

The study investigates which components of a neural network contribute to rapid generalization (grokking) and how stable that improvement remains during further training. By transferring internal attention and MLP weights along with token embeddings and readout, the authors achieve a 5.46‑percentage‑point boost in early accuracy and a 558‑step reduction in confirmation latency, while also demonstrating that freezing transferred representations largely prevents post‑grokking relapse. The work delineates clear component‑level differences between acceleration and stability, and identifies architectural limits where omitting donor embeddings leads to significant performance loss.

By Zeyu Jia
arXiv Machine Learning
Oct 28

Error Diffusion Replaces Latent Weights: Training Binary-Weight Language Models on a Few Bits of Dithered State

The paper introduces an error‑diffusion (ED) optimizer that replaces the full‑precision latent copy of binary‑weight language models with a small 2‑8‑bit residual buffer, reducing training state per weight from 97 bits to 3‑9 bits—up to 19× less. ED accumulates unrealized weight updates in the buffer and flips the binary weight when the accumulated value crosses a threshold aligned with the quantization grid, enabling efficient training on large transformers with minimal loss compared to latent‑weight baselines. The method scales to models with up to 19 M parameters on enwik8 and 70 M on wikitext‑103, maintaining performance within a few percent of reference while allowing on‑device adaptation within a 275 MB memory budget.

By Pierre-Olivier Bonin
arXiv AI
Sep 25

Wiring Beats Blending: Structure-Aware Compensation for Transformer Downscaling

The paper investigates converting a large pretrained transformer (1.4 B parameters) into a smaller sibling (410 M) by studying representation alignment and parameter projection. It finds that dense weight projection destroys structure, and that a low‑budget, structure‑aware compensation—separating least‑squares function alignment from variance‑preserving rescaling—yields significant gains on token‑efficient training, outperforming subcloning and standard distillation pipelines at matched budgets.

By Ravi Satya Durga Prasad Yenugula
arXiv AI
Oct 7

NeMo-DCR: Bit-Exact Delta-Compressed Refit for Scalable Agentic RL at Trillion-Parameter Scale

NeMo-DCR is a delta‑compressed refit method that sends only the changes in a trillion‑parameter policy model, achieving bit‑exact synchronization across AWS regions. By using fixed affine mappings, XOR masks, and a relay‑tree object storage, it avoids full checkpoint transfers and cross‑cluster collectives, reducing refit times from 87.5 min to 150 s for a 1 T model at 3% change rate. The approach scales to 30 B–1 T models, delivering 12–40× speedups over traditional full‑checkpoint refits.

By Songlin Jiang, Zhiyu Li, Terry Kong, Yu Yao, Youngeun Kwon, Bernard Nguyen, Ashwath Aithal, Mario Di Francesco
arXiv AI
Sep 21

TinyCeNN-LM: Quality-Gated Conversion of Pretrained Attention with CeNN-Inspired Cellular-Recurrent Layers

TinyCeNN-LM proposes a quality‑gated post‑training conversion framework that replaces attention in pretrained language models with CeNN‑inspired cellular‑recurrent layers. The method includes bounded local processing, compact recurrent memory, routing, fusion, and an accept‑or‑rollback validation step, with three specific implementations studied. Experiments on SmolLM2‑135M and Qwen3.5‑0.8B show that the conversion can accept certain layers while rejecting others based on representation fidelity and NLL thresholds, achieving minimal perplexity changes and modest downstream accuracy retention.

By Kabeh Mohsenzadegan, Vahid Tavakkoli, Kyandoghere Kyamakya
arXiv Machine Learning
Nov 6

Muir: Linear Attention with Sensitivity-Allocated Gating and Matched-Filter Readout

Muir is a new linear‑attention architecture that consolidates gating mechanisms into a low‑rank projection and uses a matched‑filter readout to re‑weight state rows. It improves over GDN‑2 and Transformers across a wide range of model sizes, lowering perplexity, boosting commonsense accuracy, and enhancing extractive recall while maintaining comparable throughput. The design also includes a numerically stable parallel update for efficient training on large chunk sizes.

By Tushaar Gangavarapu, Gil Tabak, Clemens JS Schaefer
arXiv AI
Oct 15

FON: A Draft-Revise Dual-Organ Language Model without Sequence-Level Quadratic Attention

arXiv:2610.16301v1 Announce Type: new Abstract: Self-attention assigns two functions to one mechanism-where to read and how to mix-at the price of O(L^2) training, an O(T) decode state, and position...

By Helong Hu (Zhanjiang Campus, Guangdong Ocean University, Zhanjiang, Guangdong, China), Hongdan Pan (Maoming Administrative Service Center, Maoming, Guangdong, China), Shuiqing Hu (China Pacific Insurance)