Hugging Face Trending Papers
The role of locality and weight sharing in convolutional neural networks
Read the original on Hugging Face Trending Papers →The Flow has not summarised this story yet — read it at Hugging Face Trending Papers.
Related stories
arXiv Machine Learning
Oct 22
The role of locality and weight sharing in convolutional neural networks
The paper investigates how locality and weight sharing individually influence learning in convolutional neural networks using a teacher‑student framework. By comparing convolutional and locally connected students trained with online stochastic gradient descent, the authors derive exact dynamics in the large‑feature‑dimension limit. Key findings include that for uncorrelated inputs the convolutional student learns P‑times faster, locality reduces sample complexity, and spatial correlations diminish the advantage of weight sharing, leading to a dynamic trade‑off between the two architectures over time.
arXiv Statistics ML
5d ago
Generalization properties of the token perceptron: the role of embedding and vocabulary
The paper investigates how the geometry of token embeddings influences learning in a minimal token‑perceptron model, where a linear predictor maps sequences of embedded tokens to outputs. Using the replica method, it derives generalization errors for continuous spherical embeddings and for discrete vocabulary embeddings, revealing that learning thresholds depend on embedding dimension and that optimal learners prioritize directions of greatest variance in the embedding space. The study also quantifies storage capacity and recovery thresholds for noiseless regression and random associations.
arXiv AI
Sep 11
Teacher Geometry Shapes Learnability in Teacher-Student Networks
The paper investigates how the geometry of teacher neural networks affects the learnability of student networks in teacher‑student setups. By formalizing learnability as the success rate of reaching the global minimum, the authors identify two teacher distributions—one maximizing node dissimilarity (easy) and one minimizing it (hard)—that lead to markedly different success rates across various settings and activation functions. They analyze the loss landscape of small networks, revealing two types of suboptimal local minima (out‑of‑bounds and interior) whose attraction regions depend on teacher structure, and demonstrate that adjusting learning rates for the readout layer and inner biases can improve success rates. whyItMatters:"The study highlights that teacher geometry, often overlooked, plays a crucial role in determining how effectively a student network can learn, offering guidance for designing more realistic teacher‑student experiments."
arXiv Machine Learning
Aug 4
Distill What the Student Can See: Fisher-Projected On-Policy Distillation for Vision-Language Models
arXiv:2608. 01263v1 Announce Type: new Abstract: On-policy distillation (OPD) samples trajectories from the current student policy and minimizes token-level divergence between student and teacher next-token distributions at prefixes along those trajectories.
Hugging Face Trending Papers
Oct 6
Symmetry-Aware Feature Learning: A Polynomial Separation for Multi-Index Models
We establish a polynomial sample complexity separation between symmetry-aware and symmetry-agnostic feature learning. We study growing-rank multi-index models with high-dimensional Gaussian covariates...
arXiv Machine Learning
Oct 7
Symmetry-Aware Feature Learning: A Polynomial Separation for Multi-Index Models
arXiv:2610.08420v1 Announce Type: new Abstract: We establish a polynomial sample complexity separation between symmetry-aware and symmetry-agnostic feature learning. We study growing-rank multi-index...