arXiv:2608. 07851v1 Announce Type: new Abstract: Residual connections rely on a static residual pathway, and are essential for training deep neural networks.
arXiv:2601. 02451v2 Announce Type: replace-cross Abstract: Graph Neural Networks (GNNs) suffer from over-smoothing in deep architectures and expressiveness bounded by the 1-Weisfeiler-Leman (1-WL) test.
arXiv:2607. 18130v1 Announce Type: new Abstract: Most parameter-efficient finetuning (PEFT) methods adapt weights or activations, thus leaving one of the key Transformer components unchanged: residual connections.
The paper introduces Spectral‑Sphere‑Constrained Hyper‑Connections (s²HC), a new method for controlling the residual matrices used in Hyper‑Connections (HC). Unlike previous doubly stochastic constraints that caused identity degeneration, expressivity bottlenecks, and parameterization inefficiencies, s²HC confines these matrices to a spectral norm sphere, restoring flexibility over subdominant spectra and eliminating unstable Sinkhorn‑Knopp iterations. This approach preserves training stability while allowing expressive, non‑degenerate residual matrices.
The paper introduces a feedforward graph architecture that uses several frozen large language models as computational nodes connected through a shared continuous latent space via learned linear projections. By jointly optimizing projection matrices through backpropagation, the system combines the representations of three small frozen models with two larger ones, culminating in a lightweight cross‑attention output node. With only 17.6 M trainable parameters, the architecture attains state‑of‑the‑art results on ARC‑Challenge, OpenBookQA, and MMLU, surpassing both individual constituent models and parameter‑matched learned classifiers.
The paper introduces GRIP, an algorithm‑agnostic framework for machine unlearning in Mixture‑of‑Experts large language models. GRIP enforces hard geometric constraints on router updates, projecting gradient changes into the null space of the retain set’s routing matrix to prevent routing manipulation. Two variants—training‑time stochastic projection and post‑training analytical correction—show significant improvements in routing stability, retain accuracy, and resistance to white‑box adversarial recovery across two MoE models.
arXiv:2606. 10324v1 Announce Type: new Abstract: The analogy between deep neural network forward passes and renormalization group (RG) flows has been repeatedly noted in the literature, but existing treatments remain qualitative: depth is described as a coarse-graining scale, attention is likened to a partition function, and representations are said to flow toward fixed points.
IntBMoE introduces a block‑conditioned mixture‑of‑experts that decouples participation, execution, and materialization by combining dense expert composition with sparse block execution. Each internal layer uses a lightweight hypernetwork to merge all expert bases into a single composed expert, while a router selects only a few blocks per token, keeping compute and memory costs low. Experiments on image classification, language modeling, and sequential recommendation demonstrate consistent performance gains, and the model is deployed in AMap’s generative recommendation system, improving UVCTR by 2.4% in online A/B tests.
arXiv:2608. 16384v1 Announce Type: cross Abstract: Universal visual representations require adaptation mechanisms that adapt across heterogeneous domains without fragmenting knowledge into domain-specific modules.
The paper introduces GaugeLasso, a method that applies symmetric group‑lasso penalties to transformer channels during training, enabling entire tensor slices to be zeroed out while maintaining dense tensors for GPU efficiency. By calibrating channel penalties based on inference utility per compute, the network self‑organizes into depth‑dependent structural profiles that can be dramatically smaller than the original architecture, achieving up to 255‑fold compression on a polynomial division task and outperforming hand‑designed baselines on language modeling and autoencoding benchmarks. The approach also accelerates training and reveals over‑provisioned axes that guide subsequent design iterations.
arXiv:2606. 16112v1 Announce Type: cross Abstract: Residual architectures are ubiquitous in deep learning, but they suffer from a subtle structural limitation: the norm of the residual stream can grow rapidly with depth.
arXiv:2606. 16456v1 Announce Type: cross Abstract: Mixture-of-Experts (MoE) models enable efficient scaling, but training them from scratch remains prohibitively expensive.
arXiv:2606. 24679v1 Announce Type: cross Abstract: Data preparation pipelines improve data quality in machine learning by transforming raw tables into learning-ready data through sequential cleaning and feature transformation operators.