arXiv Machine Learning September 22, 2026 By Rithin Nagaraj, Rupa Laalasa Oruganti, Prerna Subhashchandra Kunder, Ashwini M Joshi

Comparing Latent Concept Formation in State Space Models and Transformers via Sparse Autoencoders

Read the original on arXiv Machine Learning →

The paper compares latent representations in Selective State Space Models (SSMs) like Mamba and Transformers such as Pythia using Sparse Autoencoders. Across a 10‑million token corpus, 99.98% of Mamba features align closely with Pythia’s, supporting the Universality Hypothesis that core semantic representations are similar across architectures. A tiny 0.02% of features diverge, with Mamba’s recurrent bottleneck causing it to compress syntactic anomalies into polysemantic neurons, whereas Pythia’s attention can isolate distinct formatting edge‑cases.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

arXiv AI

Sep 1

Concepts Whisper: Spectral Anti-Concentration and the Dual Geometry of Transformer Representations

arXiv:2605.01609v2 Announce Type: replace-cross Abstract: We find that transformer concept representations systematically anti-concentrate in the spectral tail of the unembedding covariance, encoding...

By Pratyush Acharya, Nuraj Rimal, Habish Dhakal

llms multimodal safety

arXiv Machine Learning

Jul 15

Hierarchical Latent Structures in Data Generation Process Unify Mechanistic Phenomena across Scale

arXiv:2603. 06592v2 Announce Type: replace-cross Abstract: Contemporary studies in mechanistic interpretability have uncovered many puzzling phenomena in the neural information processing of Transformer-based language models, such as induction heads, function vectors, and the Hydra effect.

By Jonas Rohweder, Subhabrata Dutta, Iryna Gurevych

llms safety

arXiv AI

Sep 15

LLM-Microscope: Uncovering the Hidden Role of Punctuation in Context Memory of Transformers

The paper introduces LLM-Microscope, a toolkit for measuring how Large Language Models encode contextual information at the token level. It shows that seemingly minor tokens—such as determiners, stopwords, and punctuation—carry surprisingly high contextual weight, and removing them degrades performance on benchmarks like MMLU and BABILong-4k. The study also finds a strong link between contextualization and linearity, indicating that the transformation between layers can be approximated by a single linear mapping when tokens are well contextualized.

By Anton Razzhigaev, Matvey Mikhalchuk, Temurbek Rahmatullaev, Elizaveta Goncharova, Polina Druzhinina, Ivan Oseledets, Andrey Kuznetsov

llms rag benchmarks

arXiv AI

Jul 28

Through the Bottleneck: How Multi-head Latent Attention Separates Content from Position in Language Models

arXiv:2607. 23054v1 Announce Type: cross Abstract: Multi-head Latent Attention (MLA), introduced in DeepSeek-V2, compresses key-value pairs through a shared low-rank bottleneck (cKV), achieving 81% KV-cache reduction during inference.

By Dhruvil S, Fenil Sojitra, Ravirajsinh Chauhan

llms fine-tuning safety

arXiv Machine Learning

Sep 10

LLM Layers Immediately Correct Each Other

arXiv:2609.07876v1 Announce Type: cross Abstract: Recent methods in language model interpretability employ techniques such as sparse autoencoders to decompose residual stream contributions into linea...

By Arjun Patrawala, Jiahai Feng, Erik Jones, Jacob Steinhardt

llms safety

arXiv AI

Jun 2

MENTIS: What Belief Changes Under Alignment? Measuring Multi-Scale Latent Torsion in Language Models

arXiv:2606. 01060v1 Announce Type: cross Abstract: Preference alignment has substantially improved the observable behavior of large language models, yet it remains unclear what alignment changes internally.

By Partha Pratim Saha, Samarth Raina, Mayur Parvatikar, Amit Dhanda, Vinija Jain, Aman Chadha, Amitava Das

llms safety

One email a morning, machine-written

Related stories

Concepts Whisper: Spectral Anti-Concentration and the Dual Geometry of Transformer Representations

Hierarchical Latent Structures in Data Generation Process Unify Mechanistic Phenomena across Scale

LLM-Microscope: Uncovering the Hidden Role of Punctuation in Context Memory of Transformers

Through the Bottleneck: How Multi-head Latent Attention Separates Content from Position in Language Models

LLM Layers Immediately Correct Each Other

MENTIS: What Belief Changes Under Alignment? Measuring Multi-Scale Latent Torsion in Language Models