The paper introduces the directional linear separability measure (D‑LSM) to quantify how much linear separability is preserved or improved by injective affine maps in neural networks. It characterizes the geometry supporting D‑LSM, proves its invariance under injective affine embeddings, and derives conditions for gated activations (ReLU, GELU, SiLU) to preserve and recover samples. Experiments validate the theoretical bounds, demonstrate affine‑tube constructions that achieve guaranteed recovery, and apply the method to Vision Transformer representations to obtain early post‑activation separability certificates.
arXiv:2606. 08721v1 Announce Type: new Abstract: Modern neural classifiers commonly rely on linear readouts, yet predictive metrics alone do not characterize the class-wise geometry of the representations on which such readouts operate.
arXiv:2510. 15814v2 Announce Type: replace-cross Abstract: Universality results for equivariant neural networks remain rare.
arXiv:2406. 08966v3 Announce Type: replace Abstract: The separation power of a machine learning model refers to its ability to distinguish between different inputs and is often used as a proxy for its expressivity.
arXiv:2608. 09707v1 Announce Type: cross Abstract: Embedding trained neural networks as surrogates within optimisation problems is an established practice in operations research.
The paper introduces Mixture of Activations (MoA), a token‑adaptive feedforward network design that mixes multiple activation functions using lightweight gates while sharing linear projections. It also presents learnable activations (LA) as an input‑independent variant. The authors theoretically prove that MoA strictly surpasses both fixed‑activation FFNs and LA in expressive power, and empirically demonstrate that MoA achieves lower loss and better scaling on dense and MoE language models from 0.12 B to 2 B parameters with minimal overhead.
arXiv:2606.28444v2 Announce Type: replace-cross Abstract: Classical universal approximation theorems (UAT) establish the expressive power of sigmoidal multilayer perceptrons, but they do not specify...
arXiv:2606. 28444v1 Announce Type: cross Abstract: Classical universal approximation theorems establish the expressive power of sigmoidal multilayer perceptrons, but they do not prescribe how initial weights should encode the geometry of a data distribution.
arXiv:2606. 16028v1 Announce Type: new Abstract: Modern deep learning architectures are increasingly multi-task and multi-modal, using a pretrained foundation model combined with task-specific, fine-tuned models.
arXiv:2607. 05546v1 Announce Type: cross Abstract: We develop a unified function space theory of deep fully connected neural networks.
The paper introduces a method for constructing L-Lipschitz deep residual networks (ResNets) using a Linear Matrix Inequality (LMI) framework. By reformulating the ResNet architecture as a pseudo-tridiagonal LMI and applying the Gershgorin circle theorem, the authors derive closed‑form constraints on network parameters that guarantee Lipschitz continuity. The work also presents a compositional framework for handling recursive systems in hierarchical architectures, while noting that the Gershgorin-based approximations can over‑constrain the system, reducing expressive capacity.
arXiv:2106. 06998v5 Announce Type: replace Abstract: Training convolutional neural networks at scale demands substantial memory, largely because intermediate activations must be stored for backpropagation.
arXiv:2607. 16295v1 Announce Type: cross Abstract: Mechanistic interpretability has made significant strides in understanding neural network representations, with sparse dictionary learning (SDL) methods, most prominently sparse autoencoders, as a central paradigm.