arXiv:2608. 06631v1 Announce Type: cross Abstract: Cryptanalytic extraction has been demonstrated for ReLU networks, for networks using componentwise activations such as GELU or SiLU, and for a Transformer's final projection matrix.
arXiv:2608.28843v1 Announce Type: cross Abstract: We show that smooth two-layer feed-forward networks (FFNs) expose an additional structural model extraction channel under a chosen-input raw-output o...
The paper investigates reconstructing fully connected neural networks by combining cryptanalytic weight-direction extraction with learning from predicted labels. It adapts ReLU extraction to Leaky ReLU, compares sign-search methods, and estimates biases and output layers, proving conditional bounds for direction recovery and sign selection. Experiments show that Leaky ReLU can recover 51–57 neurons in the deepest hidden layer of a large synthetic architecture, while ReLU recovers none, and that reconstructed models on CIFAR‑10 achieve high prediction agreement and outperform a distillation baseline on several functional metrics.
arXiv:2510.06692v3 Announce Type: replace Abstract: Deep Neural Networks (DNNs) have attracted significant attention, and their internal models are now considered valuable intellectual assets. Extrac...
arXiv:2609.21941v1 Announce Type: cross Abstract: The importance of deep neural networks (DNNs) is widely recognized, and the parameters obtained through training are regarded as valuable assets. Rec...
arXiv:2605. 15491v2 Announce Type: replace-cross Abstract: Layer pruning removes entire Transformer decoder blocks from large language models, but introduces a mismatch between the hidden state received by the next surviving layer and the distribution it was trained to process, leading to significant performance degradation.
arXiv:2607. 13047v1 Announce Type: new Abstract: Parameter decomposition (PD) decomposes neural networks into interpretable computational components that faithfully reflect the original network's operations.
arXiv:2606. 11722v1 Announce Type: cross Abstract: Finding interpretable directions in language-model representations is critical for understanding and controlling model behavior.
SHIFT-LLM is a training‑free post‑pruning correction framework that inserts a Linear Residual Adapter (LRA) at each depth‑pruned site in large language models. Each LRA preserves the original residual identity while adding a lightweight affine correction calibrated via closed‑form least‑squares regression on a small held‑out set, thereby approximating the hidden state that would have been produced by the removed block. Experiments across multiple model families and benchmarks show that SHIFT‑LLM consistently recovers accuracy lost to depth pruning, achieving gains up to +15.7 points on Llama‑3.1‑8B‑Instruct with only a few hundred calibration samples and no gradient computation.
arXiv:2606. 04238v1 Announce Type: cross Abstract: Aggressive weight quantization to 2-bit precision offers substantial throughput and memory gains for large language model (LLM) inference, but typically incurs severe accuracy degradation.
The paper introduces Weight-Redundancy Pruning (WRP), a forward‑free depth‑pruning technique for large language models that estimates inter‑layer redundancy using only checkpoint weights. WRP compares attention outputs and MLP down‑projection weights across layers, combining pairwise similarities with relative projection‑scale information to guide layer grouping and block selection. Experiments show that WRP consistently outperforms existing forward‑free magnitude pruning methods and approaches the performance of activation‑based pruning across various pruning settings, model families, and downstream tasks.
The paper introduces the directional linear separability measure (D‑LSM) to quantify how much linear separability is preserved or improved by injective affine maps in neural networks. It characterizes the geometry supporting D‑LSM, proves its invariance under injective affine embeddings, and derives conditions for gated activations (ReLU, GELU, SiLU) to preserve and recover samples. Experiments validate the theoretical bounds, demonstrate affine‑tube constructions that achieve guaranteed recovery, and apply the method to Vision Transformer representations to obtain early post‑activation separability certificates.
arXiv:2609.37972v1 Announce Type: cross Abstract: As Graph Neural Networks (GNNs) are widely deployed as Machine Learning-as-a-Service (MLaaS) APIs, model stealing attacks have emerged as a critical...