arXiv:2503. 06473v5 Announce Type: replace-cross Abstract: Growing evidence suggests that layer attention mechanisms, which enhance interaction among layers in deep neural networks, have significantly advanced network architectures.
arXiv:2512.20636v2 Announce Type: replace-cross Abstract: Many self-attention sublayers in large language models (LLMs) can be removed with little to no loss. We attribute this to the Attention Suppr...
arXiv:2608.06411v2 Announce Type: replace-cross Abstract: Multimodal large language models (MLLMs) achieve strong performance across diverse vision-language tasks, but their efficiency is limited by...
arXiv:2610.08341v1 Announce Type: cross Abstract: Recent training-free pruning approaches for Multimodal Large Language Models (MLLMs) effectively cut computational overhead by exploiting visual redu...
arXiv:2608. 06411v1 Announce Type: new Abstract: Multimodal large language models (MLLMs) achieve strong performance across diverse vision-language tasks, but their efficiency is limited by the cost of processing numerous visual tokens.
arXiv:2507. 01900v3 Announce Type: replace-cross Abstract: Pruning is a highly effective approach for compressing large language models (LLMs), significantly reducing inference latency.
arXiv:2604. 00757v2 Announce Type: replace-cross Abstract: Large Vision Language Models show impressive performance across image and video understanding tasks, yet their computational cost grows rapidly with the number of visual tokens.
arXiv:2602. 03681v2 Announce Type: replace-cross Abstract: The quadratic computational complexity of softmax transformers has become a bottleneck in long-context scenarios.
arXiv:2601.06787v2 Announce Type: replace Abstract: Large Language Models (LLMs) are known to contain significant redundancy, yet a systematic explanation for why certain components, particularly in...
The paper introduces STD, a hierarchical token pruning framework for Large Vision‑Language Models that aligns pruning strategies with the functional roles of different network stages. By using high‑frequency spectral analysis in shallow layers, Gaussian‑smoothed attention in intermediate layers, and a stability‑adaptive trigger in deep layers, STD preserves essential visual information while aggressively reducing token counts. Experiments demonstrate that STD outperforms existing pruning methods, achieving up to 94.4% token reduction and a 3.9× speed‑up on LLaVA‑NeXT‑7B.
arXiv:2512.16391v2 Announce Type: replace-cross Abstract: Attention is the dominant source of latency during long-context LLM inference, an increasingly popular workload with reasoning models and RAG...
arXiv:2602.04852v3 Announce Type: replace Abstract: Linear attention offers a computationally efficient yet expressive alternative to softmax attention. However, recent empirical results indicate tha...
The paper introduces Multi-Stream Attention Residuals (MSAR), an extension of Attention Residuals that applies multi-head depth attention inspired by multi-head sequence attention. MSAR uses multi-scale causal convolutions to provide diverse contextual views for each head, then adaptively mixes the retrieved streams into a single-stream input, avoiding persistent multi-stream storage. Across 12 downstream benchmarks at 18B and 28B MoE scales, MSAR outperforms prior residual variants and achieves a 1.58× compute speedup over the standard residual baseline.