← Back to all news
arXiv Machine Learning By Tido Specht, Elias Benedict Krey, Nils Neukirch, Nils Strodthoff

Beyond Linear Concepts: Discovering and Aligning Non-Linear Concept Manifolds in Large Language Models

Read the original on arXiv Machine Learning →

The paper extends mechanistic interpretability of large language models by modeling concepts as low‑dimensional non‑linear manifolds rather than linear subspaces. It introduces a concept‑based alignment (CBA) score to compare these manifolds across layers and models, revealing block structures in intermediate layers, a shift from syntax‑dominated to mixed syntactic‑semantic concepts, and training‑dependent multilingual sharing. The study also shows that alignment patterns differ across model families and training stages, with adjacent stages aligning more closely than distant ones.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

Related stories

arXiv AI
Aug 26

Discovering Cross-Language Reasoning Invariance in LLMs with Geometry-Invariant Sparse Autoencoders

The study investigates whether multilingual language models solve mathematical problems using shared or language‑specific internal representations. By replaying reasoning traces across six languages and applying Centered Kernel Alignment, the authors identify layers with cross‑language alignment and train two sparse autoencoders— a baseline and a new Geometry‑Invariant SAE (GI‑SAE) that adds an InfoNCE loss to encourage language‑agnostic activations. Functional interchangeability tests reveal that higher geometric similarity does not always translate to interchangeable features, and cross‑language sharing varies by model and depth, with GI‑SAE amplifying existing structure in some models but not others.

By Igor Bogdanov, Changcheng Huang
llmssafety
More like this →