arXiv Machine Learning August 10, 2026 By Rahul Vashisht, Harish G. Ramaswamy

Faster Query-Key Learning Sharpens Attention in Self-Attention Models

Read the original on arXiv Machine Learning →

arXiv:2608. 06776v1 Announce Type: new Abstract: A standard self-attention layer consists of two interacting circuits: the query-key circuit that governs attention allocation, and the output-value circuit that maps attended representations to predictions.

Summary generated by The Flow from the publisher's feed. The full article lives at arXiv Machine Learning.

safety

arXiv Machine Learning

Jul 9

Nectar: Neural Estimation of Cached-Token Attention via Regression

arXiv:2605. 09778v2 Announce Type: replace Abstract: Evaluating softmax attention over a fixed long context requires reading every cached key-value pair for each new query token.

By Jo\~ao Monteiro, Michal Klein, Pierre Ablin, Marco Cuturi

benchmarks

arXiv Machine Learning

Jun 2

Don't Read Everything: A Curvature-Conditioned Query for Linear Attention

arXiv:2606. 01294v1 Announce Type: cross Abstract: Linear attention reduces the quadratic cost of softmax attention by maintaining a recurrent fast-weight state, but it consistently lags on in-context retrieval and long-context tasks.

By Dong Le, Thong Nguyen, Cong-Duy Nguyen, Anh Tuan Luu

arXiv Machine Learning

Jul 16

Self-Improving is Often Sudden: Enlightenment-style Finetuning for Large-Scale Models

arXiv:2607. 13395v1 Announce Type: new Abstract: The pursuit of autonomously self-improving models has attracted growing interest in the era of large-scale foundation models.

By Jing-Xiao Liao, Tianwei Zhang, Yu-Hao Jiang, Feifei Zhang, Hang-Cheng Dong, Feng-Lei Fan

llms fine-tuning multimodal benchmarks

arXiv AI

Jun 18

Dual Dimensionality for Local and Global Attention

arXiv:2606. 18587v1 Announce Type: cross Abstract: Decoder-only Transformers compute attention over the KV cache of preceding tokens.

By Zhiyuan Wang, Xuan Luo, Sirui Zeng, Xifeng Yan

llms fine-tuning efficiency

arXiv AI

Jul 23

LISA: Linear-Indexed Sparse Attention for Efficient Long-Context Reasoning

arXiv:2607. 19358v1 Announce Type: new Abstract: Recent advances in long chain-of-thought reasoning models such as DeepSeek-R1 have led to increasingly longer inference context lengths under the test-time scaling paradigm.

By Yu Zhao, Zekun Zhang, Fan Jiang, Bo Zeng, Linlong Xu, Shimin Shan, Yu Liu, Longyue Wang, Weihua Luo

efficiency benchmarks

arXiv Machine Learning

Jun 3

Neural Attention Search Linear: Towards Adaptive Token-Level Hybrid Attention Models

arXiv:2602. 03681v2 Announce Type: replace-cross Abstract: The quadratic computational complexity of softmax transformers has become a bottleneck in long-context scenarios.

By Difan Deng, Andreas Bentzen Winje, Lukas Fehring, Marius Lindauer

llms

Related stories

Nectar: Neural Estimation of Cached-Token Attention via Regression

Don't Read Everything: A Curvature-Conditioned Query for Linear Attention

Self-Improving is Often Sudden: Enlightenment-style Finetuning for Large-Scale Models

Dual Dimensionality for Local and Global Attention

LISA: Linear-Indexed Sparse Attention for Efficient Long-Context Reasoning

Neural Attention Search Linear: Towards Adaptive Token-Level Hybrid Attention Models