← Back to all news
arXiv AI By Chenhui Hu, Pengfei Cao, Kang Liu, Jun Zhao

Representational Atoms in Neural Networks

Read the original on arXiv AI →

The paper introduces fundamental representational units (FRUs) in neural networks and proposes two evaluation criteria—fidelity and stability—to assess them. It demonstrates that threshold-activated sparse autoencoders (TSAEs) can identify FRUs with high fidelity and stability, supported by large-scale experiments. The study further shows that more stable FRUs are more monosemantic and can be combined across layers to reconstruct semantic structures that align with human conceptual organization.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

Related stories

arXiv Machine Learning
Sep 22

Comparing Latent Concept Formation in State Space Models and Transformers via Sparse Autoencoders

The paper compares latent representations in Selective State Space Models (SSMs) like Mamba and Transformers such as Pythia using Sparse Autoencoders. Across a 10‑million token corpus, 99.98% of Mamba features align closely with Pythia’s, supporting the Universality Hypothesis that core semantic representations are similar across architectures. A tiny 0.02% of features diverge, with Mamba’s recurrent bottleneck causing it to compress syntactic anomalies into polysemantic neurons, whereas Pythia’s attention can isolate distinct formatting edge‑cases.

By Rithin Nagaraj, Rupa Laalasa Oruganti, Prerna Subhashchandra Kunder, Ashwini M Joshi
llmssafety
More like this →
arXiv AI
Sep 2

Are You Thinking What I am Thinking? : Examining Conceptual Separation in Neural Architectures

The paper investigates whether neural networks exhibit conceptual separation, meaning that examples of the same concept cluster together and related concepts are closer in representation space. Using geometric and distributional analyses, the authors find that Convolutional Neural Networks (CNNs) produce coherent, semantically ordered representations for familiar ImageNet concepts, but this coherence weakens for unseen concepts and under domain shift. Large Language Models (LLMs) keep distinct domains well separated, bring related subdomains closer, yet lose distinction between ambiguous topics at both mean and covariance levels.

By Jaee Ponde, Roshni Agarwal, Subhashis Banerjee
llms
More like this →
arXiv Machine Learning
Oct 28

FlowGenesis: Representation-Driven Self-Organization of Neural Architectures

FlowGenesis is a framework that lets neural networks self‑organize their architecture during training by adding or removing neurons based on a local statistic called Representation Flux. Neurons with high Flux are split to preserve function, while low‑Flux, low‑activity neurons are pruned. Experiments on image and text classification show that networks starting from different sizes converge to similar neuron counts and performance, and the method also improves continual learning by reducing parameters for new tasks without sacrificing accuracy.

By Maksim A Kazanskii
efficiency
More like this →