The paper introduces Residual-Aware Neural Spectral Capacity (ResNSC) for Transformers, enhancing the original NSC heuristic by incorporating residual updates into capacity calculations. ResNSC-DP jointly allocates residual and feed‑forward widths under a resource budget, providing certified bounds on the best achievable ResNSC score. Experiments on 121M and 374M models show that architectures selected by ResNSC-DP achieve lower perplexity than uniform or tapered allocations, and in LLaMA‑7B pruning ResNSC-DP improves average accuracy by 3.28–5.46 percentage points across eight tasks.
By Chenyu Zhu, Ruoyu Zhao, Zhichao Lu
The study shows that tabular foundation models (TFMs) can be dramatically simplified without losing performance. By replacing up to 96% of transformer blocks with a label‑free affine translator, researchers found that half‑depth models still outperform full‑depth TFMs and tuned GBDTs across 171 classification datasets. The simplification reduces parameters by up to 1.5 B, speeds GPU inference up to 6×, and cuts memory usage by 6× with minimal impact on AUROC.
The paper shows that Tabular Foundation Models (TFMs) are highly overparameterized, with up to 96% of transformer blocks removable without significant loss in performance. Using a closed‑form, label‑free affine translator, the authors demonstrate that halving the depth of eight leading TFMs costs at most $0.002$ median AUROC, while a single‑block TabPFN‑3 remains competitive. Removing depth saves up to 1.5 B parameters, speeds GPU inference up to 6×, and reduces memory usage by 6× with negligible accuracy loss.
By Irene Cannistraci, Julia E. Vogt
The paper introduces GaugeLasso, a method that applies symmetric group‑lasso penalties to transformer channels during training, enabling entire tensor slices to be zeroed out while maintaining dense tensors for GPU efficiency. By calibrating channel penalties based on inference utility per compute, the network self‑organizes into depth‑dependent structural profiles that can be dramatically smaller than the original architecture, achieving up to 255‑fold compression on a polynomial division task and outperforming hand‑designed baselines on language modeling and autoencoding benchmarks. The approach also accelerates training and reveals over‑provisioned axes that guide subsequent design iterations.
By Jed A. Duersch, Na\"im Es-Sebbani, Nathana\"el Haas, Zied Bouraoui
Spectral-MoE introduces a method for allocating heterogeneous per-layer expert utilization in dense-to-mixture-of-experts (MoE) conversion of large language models. It offers both static and dynamic allocators that learn to distribute a utilization budget across layers, improving performance over uniform allocation in both analytic- and learning-based conversion baselines. Experiments on Llama-2-7B, Llama-3-8B, and Qwen2.5-7B show significant reductions in perplexity and accuracy gains at various FFN utilizations.
By Zachary Broveak, Christian Kubicka, Thierry Tambe
LILA (Latent-Informed Layer Analysis) introduces a calibration‑free method for structured pruning of large language models by scoring neuron importance using the Kolmogorov–Smirnov distance between singular value distributions of full and neuron‑ablated feed‑forward network weight matrices. The approach requires no training, calibration data, or auxiliary networks, and outperforms existing methods such as PruneNet and SliceGPT on LLaMA‑2‑7B and Phi‑2 at various sparsity levels. After a single epoch of LoRA fine‑tuning, LILA matches heavily calibrated baselines, and a Neural Tangent Kernel analysis provides theoretical support for its spectral importance criterion. Additionally, LILA can dynamically allocate sparsity budgets, achieving state‑of‑the‑art generative preservation and revealing architectural bottlenecks at higher compression.
By Sankar Behera, Dhruv Singh, Anshika Agnihotri, Raj Kumar Choudhary, Satyadev Ahlawat, Yamuna Prasad