The paper investigates how diffusion models learn and generalize algorithmically, focusing on a modular addition task with images. In a single‑image setting, the model encodes operands as periodic representations and combines them via self‑attention and feed‑forward layers that mirror trigonometric addition identities. In a diverse‑image setting, the sampling process separates into an arithmetic computation phase and a visual refinement phase, with the transition marked by an internal frequency‑entropy signal.
By Joon Hyeok Kim, Yong-Hyun Park, Mattis Dals{\ae}tra {\O}stby, Jiatao Gu
arXiv:2606. 28122v1 Announce Type: cross Abstract: Neural operators provide deep neural networks for learning mappings between function spaces.
By Alex Colagrande, Paul Caillon, Eva Feillet, Alexandre Allauzen
arXiv:2603.23784v1 Announce Type: cross
Abstract: Grokking-the phenomenon where validation accuracy of neural networks on modular addition of two integers rises long after training data has been memo...
By Anand Swaroop
arXiv:2606. 01372v1 Announce Type: cross Abstract: Can neural networks learn abstract algebraic rules, or do they merely memorize training patterns?
By Divyansh Jha, Yuanfang Xie, Varan Mehra, Brennen Yu
arXiv:2606. 03645v1 Announce Type: cross Abstract: Large Language Models exhibit paradoxical fragility in fundamental arithmetic, implying a disconnect between internal computation and discrete output.
By Liuyuan Wen, Xun Zhu, Lihao Huang, Wenbin Li, Yang Gao
The paper introduces a transformer‑based sequence‑to‑sequence model with dynamic batching that automatically simplifies complex expressions involving multiple elliptic Gamma functions. It learns to apply algebraic identities, especially SL(2,ℤ) and SL(3,ℤ) modular transformations, to reduce scrambled expressions to canonical forms. Experiments show over 99% accuracy on in‑distribution tests and more than 90% accuracy under significant extrapolation, indicating the model internalizes the underlying algebraic rules rather than merely memorizing patterns.
By Yi Fan, Vishnu Jejjala, Yang Lei
arXiv:2607. 07066v1 Announce Type: cross Abstract: Transformers have demonstrated a remarkable ability to learn algorithmic reasoning, yet mechanistic analyses have mostly focused on globally invertible operations such as cyclic addition and group composition.
By Zitong Andrew Chen, Junaid Hasan, Akhil Srinivasan, Hemkesh Bandi, Jarod Alper
arXiv:2410. 24050v3 Announce Type: replace Abstract: Large-scale pretraining of transformers has been central to the success of foundation models.
By Ambroise Odonnat, Wassim Bouaziz, Vivien Cabannes
The paper introduces CAFE+FNO, a Fourier Neural Operator variant that enhances kernel generation by combining Fourier–Chebyshev features through parallel affine branches and a Hadamard product. This method produces frequency interactions explicitly, allowing a kernel MLP to map each normalized spectral coordinate to a complex channel‑mixing matrix while keeping the number of trainable parameters independent of the number of modes. Experiments on five PDE benchmarks and ablation studies demonstrate the effectiveness of the proposed basis configuration, multiplicative composition, and bandwidth learnability.
By Hyungjoon Juen, Minwoo Shin
Mode Dynamics is a framework that links coordinated parameter motion during neural network training to changes in model outputs. By decomposing recorded parameter trajectories into collective modes and mapping each mode to its state‑dependent output effect, the framework uses decomposition, reconstruction, and concentration mechanisms to analyze how behavior emerges from parameter updates. Experiments on CIFAR‑100 autoencoding and a Pythia‑70M pretraining trajectory show that a small set of leading modes can reconstruct behavior, that induced effects depend on the training path, and that modal complexity varies with task and data.
arXiv:2604. 11613v4 Announce Type: replace-cross Abstract: Transformers can perform in-context classification from a few labeled examples, yet the inference-time algorithm remains opaque.
By Patrick Lutz, Themistoklis Haris, Arjun Chandra, Aditya Gangrade, Venkatesh Saligrama
Euclidean Fourier Neural Operators (EFNOs) extend Fourier neural operators by making the spectral kernel a continuous function of physical wavevectors, thereby removing dependence on specific periodic domain shapes and sizes. This domain‑independent formulation allows EFNOs to learn operators that generalize across different grid resolutions and domain geometries. Experiments on a heat equation and a materials‑science task demonstrate that EFNOs can successfully transfer learned mappings to unseen grid sizes and crystal structures.
By Nathanael Bosch, Niklas Frederik Schmitz, Michael F. Herbst