arXiv Machine Learning
RoPE attention is an exact forward-pass gradient step with softmax intact
Read the original on arXiv Machine Learning →The Flow has not summarised this story yet — read it at arXiv Machine Learning.
Related stories
arXiv Machine Learning
Jun 2
Don't Read Everything: A Curvature-Conditioned Query for Linear Attention
arXiv:2606. 01294v1 Announce Type: cross Abstract: Linear attention reduces the quadratic cost of softmax attention by maintaining a recurrent fast-weight state, but it consistently lags on in-context retrieval and long-context tasks.
arXiv Machine Learning
3d ago
Exact Finite Attention Responses From RoPE Derivatives
arXiv:2609.14127v1 Announce Type: cross Abstract: We derive exact local responses for attention interventions, allowing candidate edits to be scored from a cached baseline and one backward pass. The...
arXiv Machine Learning
Aug 5
Provably Learning Multi-Head Attention with Queries
arXiv:2608. 03294v1 Announce Type: new Abstract: We study the problem of learning multi-head softmax attention from black-box input-output access.
arXiv AI
Jul 22
Soft-TransFormers for Continual Learning
arXiv:2411. 16073v4 Announce Type: replace-cross Abstract: Inspired by the Well-initialized Lottery Ticket Hypothesis (WLTH), we introduce Soft-TransFormers (Soft-TF), a continual learning framework that adapts a frozen pre-trained Transformer through task-specific soft subnetworks: real-valued multiplicative masks over the query, key, value, and output projections of selected self-attention layers.
Hugging Face Trending Papers
Sep 9
EFQ-Softmax: Exp-Free Quantization for Softmax
Low-bit attention accelerates Transformer inference by moving the $QK^\top$ and $PV$ matrix multiplications to FP8 or FP4 matrix engines. However, the softmax path often evaluates shifted-score expone...
arXiv Machine Learning
Aug 20
L\'evy Attention: Single-Pass Predictive Uncertainty for Continuous-Time Attention
arXiv:2608. 19171v1 Announce Type: new Abstract: Deep models for irregularly-sampled time series answer queries at arbitrary continuous timestamps, yet report nothing about how far each answer should be trusted.