arXiv:2607. 19456v1 Announce Type: cross Abstract: We derive four memory-optimal inference artifacts for transformer attention using the Mathematics of Arrays (MoA), each following directly from the forward-pass Denotational Normal Form (DNF) of with the query-row index fixed to the current decode step.
arXiv:2606. 07713v1 Announce Type: cross Abstract: The attention mechanism is the dominant computational bottleneck in modern transformer-based AI.
arXiv:2607. 14568v1 Announce Type: cross Abstract: A companion study ran a 35B mixture-of-experts model on a 2011 NVIDIA Tesla C2075 (Fermi, sm_20, 6GB) as a GPU-prefill/CPU-decode hybrid, because the 4-bit model did not fit in device memory (arXiv:2606.
arXiv:2607. 15456v1 Announce Type: new Abstract: Looped, weight-tied Transformers reduce parameters by reusing a block, but decoding still stores a separate K/V cache for every recurrence step.
arXiv:2605. 15250v3 Announce Type: replace-cross Abstract: Multi-head Latent Attention (MLA), the attention used in DeepSeek-V2/V3, jointly compresses keys and values into a low-rank latent and matches the H100 roofline almost perfectly.
Stream-CQSA is an attention-level out‑of‑memory recovery framework that uses cyclic quorum set (CQS) decomposition to recursively split an infeasible attention call into independent subsequence tasks. Each task is executed with a compatible inner kernel and the local statistics are recomposed to recover the full attention output exactly, whether the wrapped kernel is exact or approximate. Compared with FlashAttention‑2, Stream‑CQSA achieves comparable 16‑bit forward‑output error and matches backward‑gradient error when FlashAttention‑2 fits in GPU memory, but it incurs higher runtime and continues to produce outputs beyond FlashAttention‑2’s sequence‑length boundary where FlashAttention‑2 OOMs. whyItMatters":"Stream‑CQSA turns memory‑capacity failures into recoverable executions, enabling large‑context language models to run beyond the limits of existing attention implementations without sacrificing correctness."
The paper introduces Right In-Place (RiP) convolution, a memory‑efficient strategy that corrects and generalizes previous in‑place convolution formulations to arbitrary stride, dilation, padding, and rectangular kernels. RiP aligns each layer’s input and output within a shared workspace, enabling safe, row‑major access with minimal memory overhead. Experiments on 10,000 random layers and 84 layers from 25 architectures show no corruption, matching or improving on existing herringbone workspaces while reducing memory usage by up to 24.8% and lowering peak activation memory on Raspberry Pi Pico MCUs by 12.5–33.3% without affecting cycle counts.
arXiv:2605. 01910v2 Announce Type: replace-cross Abstract: Autoregressive decoding becomes bandwidth-limited at long contexts, as generating each token requires reading all $n_k$ key and value vectors from KV cache.
arXiv:2606. 20005v1 Announce Type: cross Abstract: Attention distillation, which trains one attention distribution to match another by minimizing their Kullback-Leibler (KL) divergence, is widely used in knowledge distillation, model compression, continual learning, and sparse-attention LLM training.
arXiv:2606. 13392v1 Announce Type: new Abstract: Ultra-long-context capability is becoming indispensable for frontier LLMs: agentic workflows, repository-scale code reasoning, and persistent memory all require the model to jointly attend over hundreds of thousands to millions of tokens, yet the quadratic cost of softmax attention makes this untenable at deployment scale.
The paper argues that for large‑context autoregressive language‑model inference, memory bandwidth—specifically the Key‑Value (KV) cache—becomes the limiting resource rather than arithmetic throughput. It analytically derives how arithmetic intensity decays with context length for NVIDIA H100, NVIDIA B200, and AMD MI300X, identifies crossover points where KV traffic overtakes weight traffic, and evaluates representative techniques across five compression domains. The study finds a three‑regime behavior: below the crossover, weight traffic dominates and KV compression offers little benefit; beyond it, KV traffic dominates and compression methods trade quality for bandwidth, with paging and prefix sharing being lossless but capacity‑limited, while quantization and eviction directly reduce bandwidth at the cost of accuracy. whyItMatters":"The work provides a unified analytical framework and a standardized protocol that enable consistent comparison of KV‑compression techniques across hardware and workloads, guiding practitioners in selecting appropriate methods for long‑context inference."
arXiv:2606. 16352v1 Announce Type: cross Abstract: Computation integrity of remote large language model (LLM) serving can be questionable.
arXiv:2608. 15602v1 Announce Type: cross Abstract: While binary quantization theoretically promises extreme compression and acceleration for Large Language Models (LLMs), existing research often overlooks the necessity of specialized hardware kernels, thus failing to unleash the full acceleration potential due to persistent reliance on expensive floating-point arithmetic or runtime dequantization overheads.