The paper investigates how locality and weight sharing individually influence learning in convolutional neural networks using a teacher‑student framework. By comparing convolutional and locally connected students trained with online stochastic gradient descent, the authors derive exact dynamics in the large‑feature‑dimension limit. Key findings include that for uncorrelated inputs the convolutional student learns P‑times faster, locality reduces sample complexity, and spatial correlations diminish the advantage of weight sharing, leading to a dynamic trade‑off between the two architectures over time.
By Francesco Mori, Melanie Weber
The paper investigates how the geometry of token embeddings influences learning in a minimal token‑perceptron model, where a linear predictor maps sequences of embedded tokens to outputs. Using the replica method, it derives generalization errors for continuous spherical embeddings and for discrete vocabulary embeddings, revealing that learning thresholds depend on embedding dimension and that optimal learners prioritize directions of greatest variance in the embedding space. The study also quantifies storage capacity and recovery thresholds for noiseless regression and random associations.
By Dario Bocchi, Matteo Negri, Flavio Nicoletti, Francesco D'Amico
The paper investigates how the geometry of teacher neural networks affects the learnability of student networks in teacher‑student setups. By formalizing learnability as the success rate of reaching the global minimum, the authors identify two teacher distributions—one maximizing node dissimilarity (easy) and one minimizing it (hard)—that lead to markedly different success rates across various settings and activation functions. They analyze the loss landscape of small networks, revealing two types of suboptimal local minima (out‑of‑bounds and interior) whose attraction regions depend on teacher structure, and demonstrate that adjusting learning rates for the readout layer and inner biases can improve success rates.
whyItMatters:"The study highlights that teacher geometry, often overlooked, plays a crucial role in determining how effectively a student network can learn, offering guidance for designing more realistic teacher‑student experiments."
By Kai J. Sandbrink, Flavio Martinelli, Alexander van Meegen, Wulfram Gerstner, Johanni Brea
arXiv:2608. 01263v1 Announce Type: new Abstract: On-policy distillation (OPD) samples trajectories from the current student policy and minimizes token-level divergence between student and teacher next-token distributions at prefixes along those trajectories.
By Leyan Xue, Feng Xiong, Mingjun Ma, Changqing Zhang
We establish a polynomial sample complexity separation between symmetry-aware and symmetry-agnostic feature learning. We study growing-rank multi-index models with high-dimensional Gaussian covariates...
arXiv:2610.08420v1 Announce Type: new
Abstract: We establish a polynomial sample complexity separation between symmetry-aware and symmetry-agnostic feature learning. We study growing-rank multi-index...
By Jivan Waber, Vanessa Piccolo, Yatin Dandi, Florent Krzakala
We study what spectral summaries and scalar learning curves reveal about centered quadratic networks. We construct analytic positive input densities with exactly Gaussian one- and two-coordinate margi...
arXiv:2608. 06766v1 Announce Type: cross Abstract: Training changes a network's predictions while allocating task-relevant structure across its internal units.
By Tongxi Wang
arXiv:2610.23779v1 Announce Type: cross
Abstract: Understanding how stochastic gradient descent learns internal representations in wide neural networks trained on rich target functions is a central p...
By Shu Zhou, Jean Barbier, Sebastian Goldt
arXiv:2610.02798v1 Announce Type: new
Abstract: The Muon optimizer applies spectral orthogonalization to matrix-valued updates and has shown strong performance in large-scale neural network training,...
By Xuheng Li, Qiwei Di, Yuan Cao, Quanquan Gu
On-policy distillation, in which a teacher corrects samples that the student itself generates, presupposes that the two models speak the same language: identical VAE latents, matching architectures, and a common timestep grid. We ask what happens when none of this holds, as when the strongest teacher available and the student one wishes to deploy come from different model families, and find that the standard recipes have no answer: teacher latents cannot serve as targets in a foreign coordinate system, per-pixel losses against a teacher that stochastically re-draws local detail degenerate into blur or divergence, and timestep indices lose their meaning across mismatched schedules.
The paper investigates the loss landscape of shallow, bias‑free ReLU networks in a teacher–student setting, focusing on Gaussian population loss. It shows that adding a learned linear skip connection eliminates all spurious local minima with non‑negative student output weights when the student network is at least as wide as the teacher, whereas without the skip such minima persist even under overparameterization. Additionally, the study proves that the student network always learns the teacher’s feature subspace and that, in two dimensions, the effective width of an overparameterized student is bounded by twice the teacher’s width.
By Jakob Paul Zimmermann, Moritz Grillo, Andrei Balakin, Georg Loho