arXiv Machine Learning By Chenyu Zhu, Ruoyu Zhao, Zhichao Lu

Neural Spectral Capacity with Residual Context for Heterogeneous Resource Allocation within Transformers

Read the original on arXiv Machine Learning →

The paper introduces Residual-Aware Neural Spectral Capacity (ResNSC) for Transformers, enhancing the original NSC heuristic by incorporating residual updates into capacity calculations. ResNSC-DP jointly allocates residual and feed‑forward widths under a resource budget, providing certified bounds on the best achievable ResNSC score. Experiments on 121M and 374M models show that architectures selected by ResNSC-DP achieve lower perplexity than uniform or tapered allocations, and in LLaMA‑7B pruning ResNSC-DP improves average accuracy by 3.28–5.46 percentage points across eight tasks.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

arXiv Machine Learning
Sep 22

Neural Spectral Capacity: Measuring and Designing Architectures from Network Specification Alone

The paper introduces Neural Spectral Capacity (NSC), a closed‑form metric derived from the singular‑value spectrum of weight matrices that can be computed solely from a network’s architectural specification. Unlike traditional measures such as #Params and #FLOPs, NSC captures architectural structure (depth, width, head, FFN allocations) and can be evaluated without instantiating the model, data, or gradients. Using a dynamic‑programming solver (NSC‑DP), the authors demonstrate that NSC can efficiently identify architectures that outperform existing training‑free proxies across Transformer and CNN families, and achieve state‑of‑the‑art results in tasks such as WikiText‑103 and commonsense reasoning with LLaMA‑7B. whyItMatters":"NSC provides a fast, architecture‑only proxy that outperforms conventional metrics and training‑free proxies, enabling more effective design and pruning of large models without costly training or data."

By Chenyu Zhu, Ruoyu Zhao, Zhichao Lu
arXiv Computation and Language
Oct 1

Learning Functional Subspaces for Neural Network Compression

arXiv:2609.40127v1 Announce Type: cross Abstract: Modern transformers pair impressive capabilities with substantial memory and compute demands. Low-rank weight factorization reduces both while keepin...

By Massimo Bini, Anders Christensen, Stephan Alaniz, Judah Goldfeder, Ole Winther, Yann LeCun, Ravid Shwartz-Ziv, Zeynep Akata
arXiv Machine Learning
Oct 15

Spectral-MoE: Heterogeneous Per-Layer Expert Utilization in Dense-to-MoE Conversion

Spectral-MoE introduces a method for allocating heterogeneous per-layer expert utilization in dense-to-mixture-of-experts (MoE) conversion of large language models. It offers both static and dynamic allocators that learn to distribute a utilization budget across layers, improving performance over uniform allocation in both analytic- and learning-based conversion baselines. Experiments on Llama-2-7B, Llama-3-8B, and Qwen2.5-7B show significant reductions in perplexity and accuracy gains at various FFN utilizations.

By Zachary Broveak, Christian Kubicka, Thierry Tambe
arXiv AI
Jul 22

A Controlled Study of Attention-Only Transformers

arXiv:2607. 18363v1 Announce Type: cross Abstract: Feed-forward networks hold two thirds of a transformer's non-embedding parameters, yet the architecture has not received a necessity test that controls parameters, compute, and depth at once.

By Henry Ndubuaku, Karen Mosoyan, Jakub Mroz, Noah Cylich, Satyajit Kumar, Parkirat Sandhu, Roman Shemet, Justin H Lee