← Back to all news
arXiv AI By Yunji Wang, Junjie Yao, Linyu Liu, Pinyan Lu, Zhi-Qin John Xu

Probability-Signature Dynamics: Unpacking Modular Addition Learning Within Two-Layer Networks

Read the original on arXiv AI →

The paper investigates why neural networks trained on modular addition tasks develop Fourier-structured representations that enable exact generalization. Using probability signatures, the authors show that leading gradient interactions are cyclic shift operators, which are diagonalized by the discrete Fourier transform, leading to decoupled Fourier-mode dynamics that explain Fourier sparsity, frequency matching, and phase alignment. The framework also clarifies how label noise can accelerate early loss decrease by increasing conditional label collisions, and it is demonstrated to predict frequency behavior in other operators such as XOR.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

Related stories

arXiv Machine Learning
Oct 22

A Mechanistic Study of Algorithmic Generalization in Diffusion Models on Modular Arithmetic

The paper investigates how diffusion models learn and generalize algorithmically, focusing on a modular addition task with images. In a single‑image setting, the model encodes operands as periodic representations and combines them via self‑attention and feed‑forward layers that mirror trigonometric addition identities. In a diverse‑image setting, the sampling process separates into an arithmetic computation phase and a visual refinement phase, with the transition marked by an internal frequency‑entropy signal.

By Joon Hyeok Kim, Yong-Hyun Park, Mattis Dals{\ae}tra {\O}stby, Jiatao Gu
diffusion
More like this →
arXiv Machine Learning
Oct 8

Machine learning modularity

The paper introduces a transformer‑based sequence‑to‑sequence model with dynamic batching that automatically simplifies complex expressions involving multiple elliptic Gamma functions. It learns to apply algebraic identities, especially SL(2,ℤ) and SL(3,ℤ) modular transformations, to reduce scrambled expressions to canonical forms. Experiments show over 99% accuracy on in‑distribution tests and more than 90% accuracy under significant extrapolation, indicating the model internalizes the underlying algebraic rules rather than merely memorizing patterns.

By Yi Fan, Vishnu Jejjala, Yang Lei
llms
More like this →