arXiv Machine Learning
Aug 11
Latent Spectroscopy: Posterior Collapse as a Feature
arXiv:2605. 22691v2 Announce Type: replace Abstract: We show that, in linear Gaussian VAEs, posterior collapse is a form of latent feature selection.
$\beta$-VAEs as Effective Theories: Tolerance-Dependent Dimension
arXiv:2608. 10599v1 Announce Type: new Abstract: In a $\beta$-VAE, increasing the regularization strength acts as a spectral cutoff by collapsing low-utility latent coordinates.
Related stories
arXiv AI
Jul 16
Transforming Rank: How Architecture Navigates the Spectral Pathologies of Depth
arXiv:2607. 14018v1 Announce Type: cross Abstract: We investigate how each component of the Transformer feedforward block architecture design determines how much rank survives across depth at initialization.
Hugging Face Trending Papers
Jul 15
Transforming Rank: How Architecture Navigates the Spectral Pathologies of Depth
We investigate how each component of the Transformer feedforward block architecture design determines how much rank survives across depth at initialization. We reinterpret skip connections and normalization, long understood as controlling magnitude, as mechanisms for preserving gradient rank across depth, since the very matrix multiplications and nonlinear activations that make the network expressive also reduce the rank.
arXiv Machine Learning
Jul 8
$\mathbf{\lambda}$-VAE: Variance Equalization for Posterior Collapse
arXiv:2607. 05531v1 Announce Type: new Abstract: Variational Autoencoders (VAEs) frequently suffer from posterior collapse, a failure mode in which the approximate posterior converges to the prior, rendering the latent code uninformative.
arXiv AI
Jun 4
Low-Rank Decay for Grokking in Scale-Invariant Transformers: A Spectral-Geometric View
arXiv:2606. 04405v1 Announce Type: cross Abstract: Modern Transformer architectures frequently employ normalization mechanisms such as RMSNorm and Query-Key Normalization, making parts of the model approximately scale-invariant with respect to weight magnitudes.
arXiv Machine Learning
Jun 18
Transformer Geometry Observatory TGO-I: Spectral Geometry Observatory
arXiv:2606. 19249v1 Announce Type: cross Abstract: Despite the widespread adoption of Vision Transformers (ViTs) and their success across numerous computer vision applications, the fundamental understanding of their dimensional and representational geometry remains relatively underexplored.
arXiv AI
Jul 24
The Devil is in the Spectrum: Mitigating Representation Collapse in LLMs via Topologically Regularized Side-Path
arXiv:2607. 20484v1 Announce Type: new Abstract: Large Language Models (LLMs) are fundamentally limited by representation collapse, a bottleneck that severely degrades long-context performance.
arXiv Machine Learning
Jun 2
How Neural Losses Shape VAE Latents
arXiv:2606. 00635v1 Announce Type: new Abstract: Modern VAEs are rarely trained with the pointwise likelihood implied by the standard $\beta$-VAE objective.
arXiv AI
Jun 26
Beyond the Hard Budget: Sparsity Regularizers for More Interpretable Top-k Sparse Autoencoders
arXiv:2606. 27321v1 Announce Type: cross Abstract: Sparse autoencoders (SAEs) have become a leading tool for interpreting the representations of vision foundation models, decomposing their polysemantic activations into a larger set of sparse, more monosemantic features.
Hugging Face Trending Papers
Jun 17
Transformer Geometry Observatory TGO-I: Spectral Geometry Observatory
Despite the widespread adoption of Vision Transformers (ViTs) and their success across numerous computer vision applications, the fundamental understanding of their dimensional and representational geometry remains relatively underexplored. To address this gap, we introduce Transformer Geometry Observatory (TGO), a systematic framework of experiments and analysis pipelines designed to investigate the representational geometry and dynamics of Vision Transformers.
arXiv Machine Learning
Jun 5
Scaling Laws and Spectra of Shallow Neural Networks in the Feature Learning Regime
arXiv:2509. 24882v2 Announce Type: replace Abstract: Neural scaling laws underlie many of the recent advances in deep learning, yet their theoretical understanding remains largely confined to linear models.
arXiv Machine Learning
Jul 16
Gauge-Invariant, Parameter-Insensitive Regularization for Potential Recovery from Flow on Directed Graphs
arXiv:2607. 13609v1 Announce Type: new Abstract: Recovering a latent potential from observed flow on a directed graph (a discrete Poisson problem with Dirichlet boundaries) is ill-posed, and the standard fix backfires: ridge regularization shrinks toward a gauge-meaningless origin, collapsing and reversing the recovered ordering ($+0.