The paper introduces Soft-OMP and Soft-IHT, permutation‑based variants of Orthogonal Matching Pursuit and Iterative Hard Thresholding that replace the non‑differentiable argsort with continuous soft‑sort operators. These differentiable algorithms enable the construction of fully trainable neural network architectures—OMP‑Net and IHT‑Net—while preserving the core greedy sparse recovery logic. The authors show both theoretically and numerically that the soft variants approximate their hard counterparts with controllable accuracy and can be extended to structured sparse recovery by learning structure‑aware weights.
arXiv:2510. 14812v2 Announce Type: replace Abstract: Structured weight sparsity accelerates training and inference on modern GPUs, but it trails unstructured dynamic sparse training (DST) in accuracy especially at extreme sparsity.
OMP-MoE is a training‑free compression framework that prunes redundant experts in Mixture‑of‑Experts large language models by framing the problem as sparse signal reconstruction solved with Orthogonal Matching Pursuit. The method greedily selects expert contributions as dictionary atoms to minimize reconstruction error, then optimizes cross‑layer expert allocation via a water‑filling strategy, and finally introduces an adaptive inference mechanism (OMP‑MoE†) that dynamically adjusts expert activation based on energy prediction. Experiments on Qwen, DeepSeek‑V2, GPT‑OSS, and Mixtral MoE show consistent performance gains at 25‑50% pruning ratios, with Qwen3‑30B‑A3B retaining 93.3% of original performance at 50% compression while achieving significant speedups.
arXiv:2608. 06912v1 Announce Type: new Abstract: The top-$k$ operation is a fundamental building block of modern sparse computation, enabling token routing, expert activation, memory selection, and attention pruning.
The paper introduces a layerwise, decoupled approach to structurally sparsify fully connected layers in pretrained neural networks. By extracting shallow two‑layer subnetworks, normalizing inner weights, and applying a structured group penalty to each block’s outer weight matrix, the method prunes neurons sequentially and reduces layer widths. The authors prove equivalence to a joint penalty for positively homogeneous activations, and demonstrate that this decoupled formulation is more robust, offering a broader regularization range and lower catastrophic over‑pruning while preserving accuracy in classification, sparse‑recovery, PINN, and OPT‑1.3B experiments.
arXiv:2605. 09382v2 Announce Type: replace Abstract: The Linear Assignment Problem is a fundamental combinatorial optimization task where classical exact solvers ensure optimality but suffer from an $\mathcal{O}(N^{3})$ bottleneck, while recent neural approximations struggle with scalability and exactness.
arXiv:2608. 15246v1 Announce Type: cross Abstract: Sparse-view computed tomography (CT) reduces radiation dose by acquiring fewer projection views, but the resulting inverse problem is highly ill-posed and often produces severe streak artifacts.
arXiv:2503. 13051v3 Announce Type: replace Abstract: Sorting and permutation learning are key concepts in optimization and machine learning, especially when organizing high-dimensional data into meaningful spatial layouts.
arXiv:2608. 06630v1 Announce Type: new Abstract: Pruning reduces the inference cost of large language models, but existing criteria primarily preserve large activations or reconstruct layer outputs.
arXiv:2607. 20555v1 Announce Type: new Abstract: The lottery ticket hypothesis proposes that large random neural networks contain sparse subnetworks that can match the performance of dense models after comparable training.
arXiv:2609.14391v1 Announce Type: new Abstract: Compressed sensing (CS) reconstructs images from highly limited measurements, but existing deep unfolding methods are typically driven by first-order o...
arXiv:2607. 04603v1 Announce Type: cross Abstract: Infrared small target detection (IRSTD) aims to identify long distance small targets from complex infrared backgrounds, and is a fundamental task in remote sensing.
arXiv:2605. 15491v2 Announce Type: replace-cross Abstract: Layer pruning removes entire Transformer decoder blocks from large language models, but introduces a mismatch between the hidden state received by the next surviving layer and the distribution it was trained to process, leading to significant performance degradation.