arXiv Machine Learning June 8, 2026 By Zunhai Su, Hengyuan Zhang, Wei Wu, Yifan Zhang, Yaxiu Liu, He Xiao, Qingyao Yang, Yuxuan Sun, Rui Yang, Chao Zhang, Jing Xiong, Hui Shen, Keyu Fan, Weihao Ye, Chaofan Tao, Taiqiang Wu, Zhongwei Wan, Tiantian Zhang, Bowen Yan, Zhen Li, Yiming Zhang, Congkai Xie, Yulei Qian, Yuchen Xie, Yik-Chung Wu, Hongxia Yang, Ngai Wong

Attention Sink in Transformers: A Survey on Utilization, Interpretation, and Mitigation

Read the original on arXiv Machine Learning →

arXiv:2604. 10098v2 Announce Type: replace Abstract: As the foundational architecture of modern machine learning, Transformers have driven remarkable progress across diverse AI domains.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

arXiv AI

Jun 2

Generic Interpretation Approach for Transformer Models Incorporating Heterogenous Attention Structures

arXiv:2605. 27458v2 Announce Type: replace-cross Abstract: Transformer has significantly propelled the development of artificial intelligence, and certainly the development of agents as well.

By Yongjin Cui, Xiaohui Fan, Huajun Chen

llms agents

arXiv AI

Jun 30

Emergence of Minimal Circuits for Indirect Object Identification in Attention-Only Transformers

arXiv:2510. 25013v2 Announce Type: replace-cross Abstract: Mechanistic interpretability aims to reverse-engineer large language models (LLMs) into human-understandable computational circuits.

By Rabin Adhikari

llms rag benchmarks safety

arXiv Machine Learning

Jun 30

A Mechanistic Study of Transformers Training Dynamics

arXiv:2410. 24050v3 Announce Type: replace Abstract: Large-scale pretraining of transformers has been central to the success of foundation models.

By Ambroise Odonnat, Wassim Bouaziz, Vivien Cabannes

llms

arXiv AI

Aug 18

Not All Attention Is Equal: A Quantitative Survey of the EEI Trade-off

arXiv:2608. 15459v1 Announce Type: cross Abstract: Attention mechanisms have driven machine learning for a decade, from neural machine translation to language models that do general-purpose reasoning.

By Aditya Singh

llms computer-vision nlp benchmarks safety

arXiv Machine Learning

Jun 30

Generalization Analysis of Transformers in Distribution Regression

arXiv:2606. 29256v1 Announce Type: cross Abstract: In recent years, models based on the Transformer architecture have seen widespread applications and have become one of the core tools in the field of deep learning.

By Peilin Liu, Ding-Xuan Zhou

llms nlp fine-tuning

arXiv Machine Learning

4d ago

Theory on Attention Dynamics for Out-of-Distribution In-Context Learning

arXiv:2609.36448v1 Announce Type: new Abstract: Transformers have demonstrated remarkable in-context learning (ICL) capabilities, enabling them to perform new tasks without additional fine-tuning. Ho...

By Junze Deng, Daouda Sow, Sen Lin, Yingbin Liang

llms fine-tuning

One email a morning, machine-written

Related stories

Generic Interpretation Approach for Transformer Models Incorporating Heterogenous Attention Structures

Emergence of Minimal Circuits for Indirect Object Identification in Attention-Only Transformers

A Mechanistic Study of Transformers Training Dynamics

Not All Attention Is Equal: A Quantitative Survey of the EEI Trade-off

Generalization Analysis of Transformers in Distribution Regression

Theory on Attention Dynamics for Out-of-Distribution In-Context Learning