arXiv Machine Learning August 10, 2026 By Rahul Vashisht, Harish G. Ramaswamy

Faster Query-Key Learning Sharpens Attention in Self-Attention Models

Read the original on arXiv Machine Learning →

arXiv:2608. 06776v1 Announce Type: new Abstract: A standard self-attention layer consists of two interacting circuits: the query-key circuit that governs attention allocation, and the output-value circuit that maps attended representations to predictions.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

safety

arXiv Machine Learning

Jul 9

Nectar: Neural Estimation of Cached-Token Attention via Regression

arXiv:2605. 09778v2 Announce Type: replace Abstract: Evaluating softmax attention over a fixed long context requires reading every cached key-value pair for each new query token.

By Jo\~ao Monteiro, Michal Klein, Pierre Ablin, Marco Cuturi

benchmarks

arXiv Machine Learning

Jun 2

Don't Read Everything: A Curvature-Conditioned Query for Linear Attention

arXiv:2606. 01294v1 Announce Type: cross Abstract: Linear attention reduces the quadratic cost of softmax attention by maintaining a recurrent fast-weight state, but it consistently lags on in-context retrieval and long-context tasks.

By Dong Le, Thong Nguyen, Cong-Duy Nguyen, Anh Tuan Luu

arXiv Computation and Language

Sep 14

SAS: Simple Attention Sparsification via End-to-End Optimization of Context Ranking

arXiv:2609.13141v1 Announce Type: new Abstract: Post-training attention sparsification reduces the quadratic cumulative attention cost of pretrained Transformers by selecting a small set of context u...

By Zhiwei Li, Lei Zhu, Hao Gu, Xiang Hu, Yan Wang, Haitao Mi, Sirui Han, Leo Liang, Zhijiang Guo

llms agents

arXiv Machine Learning

Jul 16

Self-Improving is Often Sudden: Enlightenment-style Finetuning for Large-Scale Models

arXiv:2607. 13395v1 Announce Type: new Abstract: The pursuit of autonomously self-improving models has attracted growing interest in the era of large-scale foundation models.

By Jing-Xiao Liao, Tianwei Zhang, Yu-Hao Jiang, Feifei Zhang, Hang-Cheng Dong, Feng-Lei Fan

llms fine-tuning multimodal benchmarks

arXiv AI

Sep 21

Attention-Aware Routing: Coupling Routing and Attention in MoEs

Attention-Aware Routing (AAR) augments the router in Mixture-of-Experts language models with temporal and spectral features derived from a sliding window of attention weights, thereby separating contextual information from the token’s hidden state. By keeping the base transformer frozen and training only routing parameters, AAR achieves a +3.37‑point improvement on GSM8K over a routing‑only baseline and demonstrates that routing changes propagate through the residual stream to reshape attention without directly updating the attention mechanism. The method also reduces long diverging generations, shows depth‑sensitivity affecting retrieval versus reasoning, and offers a controlled probe of routing‑relevant information across layers.

By Despoina Kosmopoulou, Anastasios Tsetsilas, Efthymios Georgiou, Giannis Karamanolakis, Swastik Roy, Alexandros Potamianos

llms fine-tuning

arXiv AI

Jun 18

Dual Dimensionality for Local and Global Attention

arXiv:2606. 18587v1 Announce Type: cross Abstract: Decoder-only Transformers compute attention over the KV cache of preceding tokens.

By Zhiyuan Wang, Xuan Luo, Sirui Zeng, Xifeng Yan

llms fine-tuning efficiency

One email a morning, machine-written

Related stories

Nectar: Neural Estimation of Cached-Token Attention via Regression

Don't Read Everything: A Curvature-Conditioned Query for Linear Attention

SAS: Simple Attention Sparsification via End-to-End Optimization of Context Ranking

Self-Improving is Often Sudden: Enlightenment-style Finetuning for Large-Scale Models

Attention-Aware Routing: Coupling Routing and Attention in MoEs

Dual Dimensionality for Local and Global Attention