The paper investigates how locality and weight sharing individually influence learning in convolutional neural networks using a teacher‑student framework. By comparing convolutional and locally connected students trained with online stochastic gradient descent, the authors derive exact dynamics in the large‑feature‑dimension limit. Key findings include that for uncorrelated inputs the convolutional student learns P‑times faster, locality reduces sample complexity, and spatial correlations diminish the advantage of weight sharing, leading to a dynamic trade‑off between the two architectures over time.
The success of convolutional neural networks is often attributed to locality and weight sharing. However, how these two properties separately shape learning, and how their effects depend on the spatia...
The paper investigates how the geometry of token embeddings influences learning in a minimal token‑perceptron model, where a linear predictor maps sequences of embedded tokens to outputs. Using the replica method, it derives generalization errors for continuous spherical embeddings and for discrete vocabulary embeddings, revealing that learning thresholds depend on embedding dimension and that optimal learners prioritize directions of greatest variance in the embedding space. The study also quantifies storage capacity and recovery thresholds for noiseless regression and random associations.
The paper investigates how the geometry of teacher neural networks affects the learnability of student networks in teacher‑student setups. By formalizing learnability as the success rate of reaching the global minimum, the authors identify two teacher distributions—one maximizing node dissimilarity (easy) and one minimizing it (hard)—that lead to markedly different success rates across various settings and activation functions. They analyze the loss landscape of small networks, revealing two types of suboptimal local minima (out‑of‑bounds and interior) whose attraction regions depend on teacher structure, and demonstrate that adjusting learning rates for the readout layer and inner biases can improve success rates. whyItMatters:"The study highlights that teacher geometry, often overlooked, plays a crucial role in determining how effectively a student network can learn, offering guidance for designing more realistic teacher‑student experiments."
arXiv:2610.08420v1 Announce Type: new Abstract: We establish a polynomial sample complexity separation between symmetry-aware and symmetry-agnostic feature learning. We study growing-rank multi-index...
We establish a polynomial sample complexity separation between symmetry-aware and symmetry-agnostic feature learning. We study growing-rank multi-index models with high-dimensional Gaussian covariates...
arXiv:2608. 06766v1 Announce Type: cross Abstract: Training changes a network's predictions while allocating task-relevant structure across its internal units.
arXiv:2608. 01263v1 Announce Type: new Abstract: On-policy distillation (OPD) samples trajectories from the current student policy and minimizes token-level divergence between student and teacher next-token distributions at prefixes along those trajectories.
arXiv:2609.24379v1 Announce Type: cross Abstract: Mechanistic interpretability of vision transformers seeks to decompose model computation into human-readable units, but learned representations entan...
arXiv:2610.23779v1 Announce Type: cross Abstract: Understanding how stochastic gradient descent learns internal representations in wide neural networks trained on rich target functions is a central p...
arXiv:2510. 24616v4 Announce Type: replace-cross Abstract: For four decades statistical physics has been providing a framework to analyse neural networks.
arXiv:2608. 13335v1 Announce Type: new Abstract: Neural networks trained by gradient descent on a smooth cost function can nevertheless learn in steps: the cost holds on long plateaus and then drops abruptly.
The paper investigates the loss landscape of shallow, bias‑free ReLU networks in a teacher–student setting, focusing on Gaussian population loss. It shows that adding a learned linear skip connection eliminates all spurious local minima with non‑negative student output weights when the student network is at least as wide as the teacher, whereas without the skip such minima persist even under overparameterization. Additionally, the study proves that the student network always learns the teacher’s feature subspace and that, in two dimensions, the effective width of an overparameterized student is bounded by twice the teacher’s width.