arXiv:2606. 09885v1 Announce Type: new Abstract: Mixture-of-Experts large language models (LLMs) scale efficiently through sparse activation, yet their deployment is fundamentally constrained by the large static parameter footprint of experts.
Sparse autoencoders (SAEs) are commonly used to interpret large language models, but their reliability after pruning is unclear. This study shows that pruning’s effect on an SAE is governed by perturbation energy, a covariance-weighted norm, and that magnitude pruning distorts the representation space by ignoring activation geometry. Activation-aware pruning methods such as Wanda and SparseGPT better preserve SAE behavior, and the authors find that middle layers are especially vulnerable, leading them to propose a layer‑wise sparsity allocation that reduces perplexity for a given sparsity level.
The paper presents a PyTorch-based framework for designing and optimizing binarized neural networks, incorporating freezing and pruning mechanisms. It introduces a novel pruning method that uses a global weighting scheme to assess parameter importance across abstraction levels, achieving a 70% pruning rate on VGG11 without sacrificing accuracy—outperforming existing binarized pruning results of 41%. The framework facilitates rapid, reproducible evaluation and prototyping of state‑of‑the‑art binarized network techniques.
arXiv:2608.30741v1 Announce Type: new Abstract: A central question in modern machine learning is how much a trained model can be compressed without changing its behavior, to reduce the memory, comput...
arXiv:2607. 10803v1 Announce Type: cross Abstract: Understanding which parameters are influential in Large Language Models (LLMs) is central to improving their efficiency, reliability, and interpretability.
arXiv:2508. 13836v2 Announce Type: replace-cross Abstract: Pruning is a core technique for compressing neural networks to improve computational efficiency.
arXiv:2504. 05349v4 Announce Type: replace-cross Abstract: Network pruning is used to reduce inference latency and power consumption in large neural networks.
arXiv:2607. 11473v1 Announce Type: cross Abstract: In this paper, we propose TECO, a multi-dimensional pruning framework to collaboratively prune the three dimensions (depth, width, and resolution) of convolutional neural networks (CNNs) for better execution efficiency on embedded hardware.
arXiv:2609.07803v1 Announce Type: new Abstract: Model pruning is widely used to compress deep neural networks, reducing memory and computational requirements with minimal impact on aggregate performa...
Deep neural networks have witnessed remarkable advancements in recent years and have become integral to various applications. However, alongside these developments, training and deployment of neural network models on embedding and edge devices face significant challenges due to limited memory and computational resources.
arXiv:2606. 12278v1 Announce Type: cross Abstract: Neural network pruning reduces model size by removing less important parameters while aiming to preserve predictive performance.
The paper discusses tensorizing neural networks by reshaping dense weight matrices into higher-order tensors and approximating them with low-rank tensor network decompositions. This approach offers promising model compression and introduces bond indices that create new latent spaces, potentially enhancing interpretability. Despite encouraging empirical results, tensorized neural networks remain underused, and the authors call for more research to address practical scaling and adoption challenges.