arXiv Machine Learning
Jul 3
From Approximation to Emergence: A Theory of Deep Learning
arXiv:2607. 01311v1 Announce Type: new Abstract: Deep learning has outgrown any single mathematical explanation.
Structure and Scale in Simplicial Sequence Modelling
arXiv:2606. 01302v1 Announce Type: new Abstract: Modern large-scale deep learning exhibits two striking empirical phenomena: behavioural scaling laws (predictable performance gains with increasing scale) and emergent mechanisms (structured internal representations and circuits in deep neural networks).
Related stories
arXiv AI
Jun 16
Adaptive inference and function vectors in deep transformers
arXiv:2606. 16694v1 Announce Type: cross Abstract: Transformers are widely used as a general-purpose substrate for learning complex correlations between a large collection of coupled variables, but their internal mechanisms have remained mysterious.
arXiv Machine Learning
Jun 30
A Mechanistic Study of Transformers Training Dynamics
arXiv:2410. 24050v3 Announce Type: replace Abstract: Large-scale pretraining of transformers has been central to the success of foundation models.
arXiv Machine Learning
Jun 19
A Unified Perspective on the Dynamics of Deep Transformers
arXiv:2501. 18322v2 Announce Type: replace Abstract: Transformers, which are state-of-the-art in most machine learning tasks, represent the data as sequences of vectors called tokens.
arXiv AI
Jun 15
DIFF-ERO: A Conformance-Aware Loss for Deep Learning in Process Mining
arXiv:2606. 14283v1 Announce Type: cross Abstract: Deep learning has driven many recent advances in process analytics, especially for predictive and prescriptive monitoring.
arXiv Machine Learning
Jun 25
Emergent Capabilities Arise Randomly from Learning Sparse Attention Patterns
arXiv:2606. 25010v1 Announce Type: new Abstract: Neural scaling laws for transformer language models predict smooth improvements in pretraining loss with increasing parameters, but downstream capabilities such as in-context learning are known to emerge abruptly past a certain model scale.
arXiv Machine Learning
Jul 14
Interpreting learning dynamics of autoencoders: Transient scaling and emerging concepts of the Ising model
arXiv:2607. 10285v1 Announce Type: new Abstract: We study how unsupervised autoencoders trained on microscopic spin configurations from the Ising model learn macroscopic, theory-relevant variables underlying the data-generating process.
arXiv Machine Learning
Jul 15
Hierarchical Latent Structures in Data Generation Process Unify Mechanistic Phenomena across Scale
arXiv:2603. 06592v2 Announce Type: replace-cross Abstract: Contemporary studies in mechanistic interpretability have uncovered many puzzling phenomena in the neural information processing of Transformer-based language models, such as induction heads, function vectors, and the Hydra effect.
arXiv AI
Jul 15
Higher Embedding Dimension Creates a Stronger World Model for a Simple Sorting Task
arXiv:2510. 18315v2 Announce Type: replace-cross Abstract: We investigate how embedding dimension affects the emergence of an internal "world model" in a transformer trained with reinforcement learning to perform bubble-sort-style adjacent swaps.
arXiv Machine Learning
Jul 1
Revisiting the Volume Hypothesis
arXiv:2606. 31282v1 Announce Type: new Abstract: Modern deep neural networks often contain far more parameters than needed to fit their training data, yet they achieve impressive generalization.
arXiv Machine Learning
Jun 25
Neural Scaling Universality: If Exponents Are Fixed, Time to Understand Coefficients
arXiv:2606. 25008v1 Announce Type: new Abstract: Neural scaling laws describe how pre-training loss decays as power laws with training time, model size, and compute.
arXiv Machine Learning
Jul 9
Network Dynamics-Based Framework for Understanding Deep Neural Networks
arXiv:2501. 02436v5 Announce Type: replace Abstract: Advancements in artificial intelligence call for a deeper understanding of the fundamental mechanisms underlying deep learning.