arXiv AI
Curvature Cryptanalysis of Smooth Transformer Feed-Forward Networks
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv AI
Aug 10
Cryptanalytic Extraction of Isolated Bias-Free GLU Feed-Forward Blocks by Antipodal Separation
arXiv:2608. 06631v1 Announce Type: cross Abstract: Cryptanalytic extraction has been demonstrated for ReLU networks, for networks using componentwise activations such as GELU or SiLU, and for a Transformer's final projection matrix.
arXiv Machine Learning
Aug 4
Geometry-Guided Layerwise FFN Width Allocation in Transformers
arXiv:2608. 02064v1 Announce Type: new Abstract: Feed-forward networks (FFNs) account for a large fraction of Transformer parameters, yet their hidden width is usually constant across depth.
arXiv AI
Jun 26
SharQ: Bridging Activation Sparsity and FP4 Quantization for LLM Inference
arXiv:2606. 26587v1 Announce Type: cross Abstract: Low-bit floating-point formats and semi-structured sparsity are increasingly supported by modern accelerators, yet combining them for LLM activation compression remains challenging: activations contain input-dependent outliers that dominate block scales in FP4 quantization, and directly applying N:M sparsity masks discards moderate values, coupling sparsification loss with quantization error.
arXiv AI
Jun 19
How Linear Is a Transformer Feed-Forward Block? Per-Block Linear Recoverability Is Learned, Not Architectural
arXiv:2606. 19379v1 Announce Type: cross Abstract: Transformer feed-forward networks (FFNs) are often treated as nonlinear stores of computation, yet how nonlinear a trained FFN block actually is has rarely been measured.
arXiv Machine Learning
Aug 27
Robust CurveMoE: Multi-Norm Adversarial Defense for Mixture-of-Experts Models via Mode Connectivity
Robust CurveMoE is a mixture‑of‑experts framework that protects neural networks against perturbations defined by multiple norm constraints. It connects norm‑specialized models through a low‑loss path, selectively expertises only influential layers, and shares the rest of the parameters across routing paths. The method introduces contribution‑guided partial updating to reduce curve‑construction cost and provides a theoretical bound on the objective gap between partial and full optimization, achieving consistent improvements in clean, norm‑specific, and Union accuracy on CIFAR‑100 and ImageNet‑100.
arXiv AI
Jun 26
CascadeFormer: Depth-Tapered Transformers Motivated by Gradient Fan-in Asymmetry
arXiv:2606. 26538v1 Announce Type: cross Abstract: Deep Transformers are composed of uniformly stacked residual blocks, yet their deepest layers often add little value.