← Back to all news
arXiv Machine Learning By Francesco Mori, Melanie Weber

The role of locality and weight sharing in convolutional neural networks

Read the original on arXiv Machine Learning →

The paper investigates how locality and weight sharing individually influence learning in convolutional neural networks using a teacher‑student framework. By comparing convolutional and locally connected students trained with online stochastic gradient descent, the authors derive exact dynamics in the large‑feature‑dimension limit. Key findings include that for uncorrelated inputs the convolutional student learns P‑times faster, locality reduces sample complexity, and spatial correlations diminish the advantage of weight sharing, leading to a dynamic trade‑off between the two architectures over time.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

Related stories

arXiv Statistics ML
5d ago

Generalization properties of the token perceptron: the role of embedding and vocabulary

The paper investigates how the geometry of token embeddings influences learning in a minimal token‑perceptron model, where a linear predictor maps sequences of embedded tokens to outputs. Using the replica method, it derives generalization errors for continuous spherical embeddings and for discrete vocabulary embeddings, revealing that learning thresholds depend on embedding dimension and that optimal learners prioritize directions of greatest variance in the embedding space. The study also quantifies storage capacity and recovery thresholds for noiseless regression and random associations.

By Dario Bocchi, Matteo Negri, Flavio Nicoletti, Francesco D'Amico
llmsrag
More like this →
arXiv AI
Sep 11

Teacher Geometry Shapes Learnability in Teacher-Student Networks

The paper investigates how the geometry of teacher neural networks affects the learnability of student networks in teacher‑student setups. By formalizing learnability as the success rate of reaching the global minimum, the authors identify two teacher distributions—one maximizing node dissimilarity (easy) and one minimizing it (hard)—that lead to markedly different success rates across various settings and activation functions. They analyze the loss landscape of small networks, revealing two types of suboptimal local minima (out‑of‑bounds and interior) whose attraction regions depend on teacher structure, and demonstrate that adjusting learning rates for the readout layer and inner biases can improve success rates. whyItMatters:"The study highlights that teacher geometry, often overlooked, plays a crucial role in determining how effectively a student network can learn, offering guidance for designing more realistic teacher‑student experiments."

By Kai J. Sandbrink, Flavio Martinelli, Alexander van Meegen, Wulfram Gerstner, Johanni Brea
More like this →