arXiv:2606. 05863v1 Announce Type: new Abstract: Grokking suggests that fitting the training data and learning a simple underlying rule may occur on different time scales.
By Hu Tan, Kuo Gai, Shihua Zhang
arXiv:2605.06240v2 Announce Type: replace-cross
Abstract: Forward-Forward (FF) training lets each layer learn from a local goodness criterion. In cumulative-goodness variants, later layers can inheri...
By Amirhossein Yousefiramandi
LayerRoPE introduces a depth‑wise magnitude and angular superposition mechanism that replaces per‑layer normalization gains with a single shared vector and depth‑conditioned scalars, reducing parameters and computational cost. Across 16 pre‑trained large language models and a 100B+ token model ladder, LayerRoPE outperforms traditional Pre‑, Post‑, and Peri‑Norm schemes, achieving comparable or better performance with significantly less compute and improved convergence up to 512 layers. The method demonstrates that widening the residual stream, rather than shrinking it, enhances depth stability and learning‑rate sensitivity, and it transfers effectively to latent and vision transformer architectures.
By Shikhar Srivastava, Christopher Kanan
Deep neural networks can exhibit periodic loss spikes known as the "Slingshot Mechanism" during long‑term training. This paper shows that such spikes arise from finite‑precision floating‑point arithmetic: when the correct‑class logit becomes much larger than the others, the gradient for that class is rounded to zero while others remain nonzero, breaking the zero‑sum constraint and creating a positive feedback loop that inflates feature and classifier norms. The authors term this Numerical Feature Inflation (NFI) and demonstrate that it explains the rapid norm growth, reappearance of gradients, and loss spikes, while also showing that partial absorption can still cause hidden norm growth without visible spikes.
By Hanqing Liu, Jianjun Cao, Yuanze Li, Zijian Zhou
arXiv:2608. 06177v1 Announce Type: new Abstract: Binary neural networks are very attractive for constrained deployment, enabling small footprint and low-power inference.
By Quentin Luquet de Saint-Germain, Massil Ait Abdeslam, Jean Pierre David
arXiv:2505. 22578v2 Announce Type: replace Abstract: The optimization of neural networks under weight decay remains poorly understood from a theoretical standpoint.
By Etienne Boursier, Matthew Bowditch, Matthias Englert, Ranko Lazic