← Back to all news
arXiv AI By Zihui Zhao, Yuanbo Tang, Yang Li

On the Sparsity-Storage-Accuracy Tradeoff in Parsimoniously Activated Dictionary Learning

Read the original on arXiv AI →

The paper revisits parsimoniously activated dictionary learning (PADL), a method that imposes a global regularization on the number of activated dictionary atoms. It shows that PADL can be reformulated as a maximum a posteriori estimation under a structured generative model, enabling derivation of generalization guarantees and an analytical tradeoff between sparsity, storage cost, and reconstruction accuracy. Leveraging this insight, the authors propose an efficient PADL algorithm that removes manual hyperparameter tuning, improves reconstruction performance on visual benchmarks, and accelerates inference for vision‑language models.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

Related stories

arXiv AI
1d ago

Beyond SAEs: Consistent Concept Discovery with Sparse Coding

The paper introduces Consistent Sparse Dictionary Learning (ConSDL), a method that replaces the linear encoder in sparse autoencoders with per‑sample sparse coding to improve dictionary learning. Synthetic experiments show ConSDL achieves near‑perfect recovery of dictionaries and coefficients, and in real vision and language tasks it boosts cross‑seed consistency by 17.6% for dictionaries and 35.9% for coefficients compared to six SAE baselines.

By Jinho Choi, Hyesu Lim, Jaegul Choo, Steffen Schneider
benchmarks
More like this →
arXiv Machine Learning
Oct 9

Stop Probing, Start Coding: Why Linear Probes and Sparse Autoencoders Fail at Compositional Generalisation

The paper argues that the linear representation hypothesis—where neural activations encode concepts as linear mixtures—breaks down under superposition, causing linear decision boundaries to lose linearity after projection. Classical sparse coding with per‑sample inference can recover latent factors, but sparse autoencoders (SAEs) amortise this inference into a fixed encoder, creating a systematic gap that persists across training sizes, latent dimensions, and sparsity levels. Experiments show that the failure of SAEs under out‑of‑distribution compositional shifts stems from poor dictionary learning rather than inference, and that a good dictionary can solve the problem at all tested scales, highlighting scalable dictionary learning as the key open challenge.

By Vit\'oria Barin-Pacela, Shruti Joshi, Isabela Camacho, Simon Lacoste-Julien, David Klindt
llmsefficiency
More like this →
arXiv AI
Oct 7

Inference and learning in sparse autoencoders as natural gradient flow

arXiv:2610.07389v1 Announce Type: cross Abstract: Sparse autoencoders are widely used to uncover interpretable features in neural networks, yet reliable recovery remains difficult when features overl...

By Hadi Vafaii, Tejas Rao, David Chanin, Thomas Fel, Jacob L. Yates, Bruno Olshausen, David Klindt, Dileep George, Miguel L\'azaro-Gredilla
More like this →
arXiv Machine Learning
Oct 12

Sparse Dictionary Learning for Measure-Valued Data via Linearized Optimal Transport: Efficient Algorithms and Sample Complexity

The paper introduces a dictionary learning framework for probability measure-valued data by embedding measures into Hilbert spaces using features derived from optimal transport geometry. It learns data‑adapted dictionaries through sparse autoencoder parameterizations, yielding fast, scalable representations of probability measures as sparse finite‑dimensional codes. The authors demonstrate the method on hyperspectral images, geometric point clouds, and document embeddings from large language models, and provide generalization bounds and covering number results for optimal transport and entropic transport map dictionaries.

By Brendan Mallery, Shuchin Aeron, James Murphy
llmsragsafety
More like this →