arXiv Machine Learning
Jul 3
From Approximation to Emergence: A Theory of Deep Learning
arXiv:2607. 01311v1 Announce Type: new Abstract: Deep learning has outgrown any single mathematical explanation.
Structure and Scale in Simplicial Sequence Modelling
arXiv:2606. 01302v1 Announce Type: new Abstract: Modern large-scale deep learning exhibits two striking empirical phenomena: behavioural scaling laws (predictable performance gains with increasing scale) and emergent mechanisms (structured internal representations and circuits in deep neural networks).
Related stories
arXiv AI
Jun 16
Adaptive inference and function vectors in deep transformers
arXiv:2606. 16694v1 Announce Type: cross Abstract: Transformers are widely used as a general-purpose substrate for learning complex correlations between a large collection of coupled variables, but their internal mechanisms have remained mysterious.
arXiv Machine Learning
Jun 30
A Mechanistic Study of Transformers Training Dynamics
arXiv:2410. 24050v3 Announce Type: replace Abstract: Large-scale pretraining of transformers has been central to the success of foundation models.
arXiv Machine Learning
Jun 19
A Unified Perspective on the Dynamics of Deep Transformers
arXiv:2501. 18322v2 Announce Type: replace Abstract: Transformers, which are state-of-the-art in most machine learning tasks, represent the data as sequences of vectors called tokens.
arXiv Machine Learning
Sep 22
Fractal basin geometry and the limits to predictability and reproducibility in deep learning
arXiv:2510.05606v2 Announce Type: replace Abstract: Fundamental limits to predictability are central to our understanding of many physical and computational systems. In deep learning, training outcom...
arXiv AI
Jun 15
DIFF-ERO: A Conformance-Aware Loss for Deep Learning in Process Mining
arXiv:2606. 14283v1 Announce Type: cross Abstract: Deep learning has driven many recent advances in process analytics, especially for predictive and prescriptive monitoring.
arXiv Machine Learning
Jun 25
Emergent Capabilities Arise Randomly from Learning Sparse Attention Patterns
arXiv:2606. 25010v1 Announce Type: new Abstract: Neural scaling laws for transformer language models predict smooth improvements in pretraining loss with increasing parameters, but downstream capabilities such as in-context learning are known to emerge abruptly past a certain model scale.
arXiv Machine Learning
Jul 14
Interpreting learning dynamics of autoencoders: Transient scaling and emerging concepts of the Ising model
arXiv:2607. 10285v1 Announce Type: new Abstract: We study how unsupervised autoencoders trained on microscopic spin configurations from the Ising model learn macroscopic, theory-relevant variables underlying the data-generating process.
arXiv Machine Learning
Jul 15
Hierarchical Latent Structures in Data Generation Process Unify Mechanistic Phenomena across Scale
arXiv:2603. 06592v2 Announce Type: replace-cross Abstract: Contemporary studies in mechanistic interpretability have uncovered many puzzling phenomena in the neural information processing of Transformer-based language models, such as induction heads, function vectors, and the Hydra effect.
arXiv Machine Learning
Aug 28
Neural Renormalization Group Flow for Percolation
The paper presents a supervised, scale‑shared neural architecture that learns a coarse‑graining rule for two‑dimensional site percolation. By recursively applying this rule, the model generates a latent field from which the crossing probability is predicted and a fine‑graining decoder reconstructs the largest‑cluster mask. Trained only on small lattices, the network extrapolates to larger systems, accurately recovers the spanning cluster, and reproduces finite‑size scaling near the critical point, demonstrating that the latent representation captures critical fluctuations and scale‑dependent flows consistent with renormalization‑group theory.
arXiv AI
Jul 15
Higher Embedding Dimension Creates a Stronger World Model for a Simple Sorting Task
arXiv:2510. 18315v2 Announce Type: replace-cross Abstract: We investigate how embedding dimension affects the emergence of an internal "world model" in a transformer trained with reinforcement learning to perform bubble-sort-style adjacent swaps.
arXiv Machine Learning
Jul 1
Revisiting the Volume Hypothesis
arXiv:2606. 31282v1 Announce Type: new Abstract: Modern deep neural networks often contain far more parameters than needed to fit their training data, yet they achieve impressive generalization.