Position Anchor Tuning (PAT) is a parameter‑efficient fine‑tuning method for pre‑trained point cloud transformers that improves inference efficiency. PAT reduces the computational cost of multi‑head attention and feed‑forward networks by using token aggregation‑expansion pairs: a token aggregation module (TAM) selects representative tokens based on 3D position anchors, and a token expansion module (TEM) propagates the learned representations back to the full token set. Combined with base‑sharing low‑rank adaptation (BSLoRA) for the TAMs, PAT achieves performance comparable to state‑of‑the‑art methods while requiring fewer trainable parameters and lower computational overhead.
arXiv:2601. 11667v2 Announce Type: replace-cross Abstract: Transformer architectures deliver state-of-the-art accuracy via dense full-attention, but their quadratic time and memory complexity with respect to sequence length limits practical deployment.
The paper investigates whether deep transformer layers require context from the residual stream to compute value vectors. It finds that allowing deeper layers to use a context‑free value vector—preserving original token information—significantly improves performance, and adding context afterward yields little extra benefit. The authors introduce Bank of Values (BoV), a lookup table of token‑specific value vectors for the last third of layers, which reduces compute and memory while matching or surpassing prior methods on large models.
arXiv:2606. 04032v1 Announce Type: cross Abstract: Transformers have become the standard solution for various AI tasks, with the query, key, and value (QKV) attention formulation playing a central role.
arXiv:2607. 02612v1 Announce Type: cross Abstract: Vision Transformers achieve strong image classification accuracy but process all image regions with nearly the same computation, even when many regions are redundant or uninformative.
arXiv:2608.23921v1 Announce Type: new Abstract: Recent Vision-Language Models encode high-resolution images into long visual token sequences, incurring prohibitive prefill costs. To compress them, ex...
arXiv:2605.26797v2 Announce Type: replace Abstract: We study Latent Recurrent Transformer (LRT), a lightweight augmentation of autoregressive transformers that reuses a high-level source-layer hidden...
arXiv:2603.26380v3 Announce Type: replace Abstract: The attention mechanism has been the core component in modern transformer architectures. However, the computation of standard full attention scales...
arXiv:2502.09245v3 Announce Type: replace Abstract: In contrast to RNNs, which compress their history into a single hidden state, Transformers can attend to all past tokens directly. However, standar...
arXiv:2605. 18848v3 Announce Type: replace Abstract: This paper introduces Exact Linear Attention (ELA), a mechanism that achieves linear computational complexity for Transformer attention by exploiting the exact decomposition property of kernel functions, thereby eliminating approximation error.
arXiv:2609.23733v1 Announce Type: new Abstract: Feed-forward visual geometry models such as the Visual Geometry Grounded Transformer (VGGT) have recently enabled direct 3D reconstruction from multi-v...
arXiv:2609.15131v1 Announce Type: cross Abstract: Multimodal large language models (MLLMs) require substantial computation to process numerous visual tokens across all transformer layers. Most method...
arXiv:2507. 01900v3 Announce Type: replace-cross Abstract: Pruning is a highly effective approach for compressing large language models (LLMs), significantly reducing inference latency.