The paper investigates how neural representations that use superposition can encode more features than their dimensionality allows, focusing on the dimensionality required to compute Boolean gates. For a single threshold layer with a Gaussian random dictionary and uniformly random sparse Boolean inputs, the authors derive precise dimension thresholds under two error criteria: one based on vanishing expected error count and another on high‑probability correctness of every output. The study reveals a geometric distinction—ball geometry for expected‑count thresholds and box geometry for joint reliability—explains the gap via shared reads, and provides explicit thresholds for conjunction, disjunction, and majority gates, including a detailed analysis of pairwise conjunction near the threshold.
arXiv:2607. 21633v1 Announce Type: new Abstract: Logic Gate Networks (LGNs) implement computation through compositions of Boolean operations, yet unlike classical Boolean circuits, existing LGNs do not reliably benefit from increased depth.
By Taegun An, Dohun kim, Haebeom Lee, Changhee Joo
arXiv:2510. 04500v3 Announce Type: replace Abstract: This work demonstrates how increasing the number of neurons in a network without increasing its total number of non-zero parameters improves performance.
By Linghao Kong, Inimai Subramanian, Yonadav Shavit, Micah Adler, Dan Alistarh, Nir Shavit
arXiv:2609.09556v1 Announce Type: new
Abstract: Neural networks can leverage feature superposition to encode more concepts than dimensions, but cross-feature interference constrains the linear access...
By Enrico Vompa
arXiv:2607. 03664v1 Announce Type: new Abstract: The Gaussian Error Linear Unit is usually motivated as the expected output of an input-dependent stochastic Bernoulli gate.
By Roberto Rossi
arXiv:2607. 04800v1 Announce Type: new Abstract: Neural networks are thought to represent concepts as directions in their activation space, and superposition lets them encode more concepts than they have dimensions.
By Francisco Ferreira da Silva, Stefan Heimersheim
arXiv:2609.06862v1 Announce Type: new
Abstract: Superposition refers to neural networks representing more features than they have dimensions. It offers a possible explanation for polysemantic neurons...
By Dai Shi, Xiaoyu Li, Andi Han, Jos\'e Miguel Hern\'andez-Lobato
The paper develops a mathematical theory of superposition in neural networks using frame theory and compressed sensing. It shows that a sparse binary vector of active features can be encoded by an overcomplete dictionary and recovered via a ReLU operation with a suitable bias. The authors prove recovery theorems for both random-support and worst-case support settings, providing high-probability guarantees for low-coherence dictionaries and a sharp criterion for sparsity levels, with explicit results for Gaussian random matrices and equiangular tight frames.
By Michael I. Ivanitskiy, John Jasper, Emily J. King, Dustin G. Mixon
arXiv:2606. 18538v1 Announce Type: new Abstract: One of the major difficulties in the mechanistic interpretability of neural networks is the occurrence of polysemanticity, which suggests that each neuron is typically responsible for multiple different tasks, impeding a clean interpretation of their function.
By Mriganka Basu Roy Chowdhury, Eric McLaughlin Weiner
arXiv:2602. 03970v3 Announce Type: replace-cross Abstract: We study the statistical behavior of reasoning probes in a stylized model of iterative computation inspired by neural algorithmic reasoning.
By Anastasis Kratsios, Giulia Livieri, A. Martina Neuman
The paper introduces MACCHIATO, a training algorithm that builds a ReLU‑MLP from partial truth‑table data while simultaneously constructing an explicit Boolean circuit over AND, OR, and XOR gates that certifies the network’s computation. The method iteratively projects residuals onto low‑dimensional Boolean classes, compiles the resulting circuit into a ReLU‑MLP, and uses logic minimization and influence‑based variable selection to achieve a six‑layer network with provable truth‑table error bounds. Experiments on synthetic random‑junta tasks show that these certified networks outperform Adam‑trained MLPs in data‑sparse or projection‑aligned regimes and complete faster than flat ESPRESSO in certain settings.
By Hrad Ghoukasian, Anastasis Kratsios
arXiv:2602.17493v2 Announce Type: replace-cross
Abstract: We develop a method for training neural networks on Boolean data in which the values at all nodes are strictly $\pm 1$, and the resulting mod...
By Veit Elser, Manish Krishan Lal