The paper investigates how spectral summaries and scalar learning curves can reveal properties of centered quadratic networks. By constructing analytic positive input densities with Gaussian one- and two-coordinate marginals, the authors show that certain teachers with a nonzero cycle invariant produce distinct factorized loss curves, while others yield identical anchored spectral measures. The study further explores the sample complexity required for noisy scalar loss queries and bounded initialization interventions, identifies conditions under which a single scalar loss curve can recover the initial predictor and teacher spectrum, and provides exact recovery guarantees for fixed dimensions and distinct teacher sums.
By Dekun Yang
arXiv:2609.39408v1 Announce Type: cross
Abstract: Population loss can remain nearly constant while a neural network learns a substantially more predictive representation. We establish this separation...
By Akash Kumar
arXiv:2608.29472v1 Announce Type: cross
Abstract: Knowledge distillation trains a small student model to reproduce the outputs of a large teacher model, and its progress is typically monitored throug...
By Kazuyuki Hara, Hideitsu Hino
arXiv:2608. 10172v1 Announce Type: new Abstract: Mechanistic interpretability explains models by identifying circuits inside them, but has no way to tell whether a circuit is a property of the model or an artifact of the method that found it.
By Ashim Dhor, Pin-Yu Chen
arXiv:2609.37745v1 Announce Type: new
Abstract: Neural scaling, in which loss falls as a power law with training, is central to large language models, and one recent proposal is that a $1/3$ exponent...
By Hyunseok Lee, Mihir Basil, Yizhou Liu, Jeff Gore
arXiv:2609. 12994v1 Announce Type: new Abstract: Heavy-tailed empirical spectral densities of neural-network weight matrices are widely used as diagnostics of implicit self-regularization, but the step complexity of heavy-tail emergence remains poorly understood.
By Zongmin Liu
arXiv:2510.22778v4 Announce Type: replace-cross
Abstract: A growing class of machine-learning objects -- covariance and Gram matrices, kernel and attention matrices, MIMO channel matrices, density op...
By Swagatam Das
arXiv:2607. 25624v1 Announce Type: new Abstract: Positive quadratic networks admit the low-rank representation f_U(x)=x^top UU^top x, where Uinmathbb{R}^{dtimes r} is identifiable only up to right orthogonal multiplication, representing a rank-r PSD matrix Q=UU^top.
By Pengcheng Cheng
arXiv:2608. 15472v1 Announce Type: cross Abstract: The problem of networked information aggregation, studied in Kearns et al.
By Ambar Pal
arXiv:2607. 19771v1 Announce Type: cross Abstract: Muon and related matrix-sign optimizers are increasingly used to pre-train large language models, but their effect on the internal geometry of individual weight matrices is not well understood.
By Jiachun Li
arXiv:2610. 09356v1 Announce Type: new Abstract: We prove global exponential (linear) convergence with an explicit rate in the rich scaling for wide two-layer linear networks trained with smooth Polyak-Lojasiewicz predictor losses.
By Stephen Y Zhang, Gabriel Peyr\'e
arXiv:2610.27010v1 Announce Type: cross
Abstract: We study robust estimation of a Gaussian precision matrix that decomposes as a sparse matrix minus a symmetric, possibly indefinite, low-rank matrix,...
By Chuansen Peng, Xiaojing Shen