The paper introduces fast and stable methods for triangular matrix inversion used in Delta-Rule linear transformers, a key component of efficient long‑context models such as Qwen3.5/3.6 and RWKV‑7. It systematically compares direct and iterative inversion algorithms, emphasizing numerical stability, computational complexity, and hardware efficiency, and validates the approaches with rigorous experiments in low‑precision settings. Benchmarks on NPUs show up to a 4.3× speed‑up over current SGLang implementations, improving layer performance while preserving model accuracy.
arXiv:2606. 06034v1 Announce Type: new Abstract: Matrix inversion in chunk-wise parallel linear attention is a major bottleneck for long-context modeling, particularly on NPUs, where forward-substitution-based methods exhibit limited parallelism and poor hardware utilization.
arXiv:2607. 04422v1 Announce Type: cross Abstract: Recent NVFP4 pretraining methods mainly target transformer linear layers, leaving optimizer states, optimizer arithmetic and attention underexplored in 4-bit pipelines.
The paper introduces 3BASiL-TM, a one‑shot post‑training method for compressing large language models using a sparse plus low‑rank decomposition. It employs a novel 3‑block ADMM algorithm to minimize layer‑wise reconstruction error and a transformer‑matching refinement that jointly optimizes sparse and low‑rank components across layers. Experiments show a >30% reduction in WikiText2 perplexity gap for LLaMA‑8B under a (2:4 sparse + 64 LR) setting and a 2.5× faster compression runtime on an A100 GPU compared to state‑of‑the‑art methods.
ResidualQuant introduces a 2‑bit residual quantization scheme for KV caches in looped Transformers, leveraging the similarity of KV states across loops. By using the final‑loop KV states as a reference and applying least‑square scaling, rotations, and loop‑wise mixed precision, the method achieves accurate quantization down to INT2. Experiments on various looped Transformer models show that ResidualQuant improves the accuracy‑memory tradeoff, retaining near BF16 accuracy while reducing KV storage by 80.7% and boosting decode throughput and batch size on an RTX 5090.
arXiv:2607. 18745v1 Announce Type: new Abstract: We study low-precision computation of C=AB with both factors quantized.
arXiv:2607. 25504v1 Announce Type: cross Abstract: Fine-grained weight pruning and activation sparsification have emerged as effective approaches for reducing the compute and memory cost of inference for Transformer models.
The paper revisits Kashin‑decomposition‑based weight quantization for large language models, introducing an improved algorithm that uses a sign‑randomized Discrete Cosine Transform (DCT) instead of a dense random orthogonal matrix. This change reduces per‑iteration cost from ≠(N^2) to ≠(N log N) and, combined with a greedy alternating‑update scheme, guarantees the four‑peak distribution needed for stable 2‑bit clustering while eliminating the need for multi‑restart k‑means. The resulting JAX pipeline, when paired with OPTQ‑style error compensation and QuIP‑style incoherence preprocessing, competes with state‑of‑the‑art quantization methods on OPT, Llama‑2, and Pythia at 4‑bit per channel, and remains numerically stable under stress configurations that cause other methods to diverge.
arXiv:2605. 26092v4 Announce Type: replace-cross Abstract: The deployment of Large Language Models (LLMs) and Vision Transformers (ViTs) on edge devices is significantly constrained by memory limitations and the critical timing bottlenecks introduced by dense Multiply-Accumulate (MAC) arrays.
arXiv:2510.25323v4 Announce Type: replace Abstract: Normalizing flows are deep generative models that enable efficient likelihood estimation and sampling through invertible transformations. A key cha...
DyRA introduces a dynamic, input‑adaptive approach to improve matrix multiplication in deep neural networks by correcting residual output errors during inference. Unlike prior methods that approximate only the weight matrices, DyRA directly optimizes low‑rank factors of the output, combining efficient structured computation with input‑dependent correction. Experiments across vision, speech, and language models show that DyRA consistently enhances the accuracy‑efficiency trade‑off, achieving a 1.5× GPU speedup for DINOv3 while reducing accuracy loss more than threefold compared to weight‑only baselines.
arXiv:2611.06828v1 Announce Type: new Abstract: Gated-delta linear attention layers such as Gated DeltaNet (GDN), Kimi Delta Attention (KDA), and GDN-2 are tuned for GPUs, where small tensor-core til...
arXiv:2607. 21446v1 Announce Type: new Abstract: Post-training quantization (PTQ) of diffusion transformers (DiTs) to W4A4 severely degrades output quality, because activations entering each linear layer contain outliers that 4-bit formats cannot represent.