Hugging Face Trending Papers July 14, 2026

Toward Localizing and Repairing Bias in Transformer Attention Heads

Read the original on Hugging Face Trending Papers →

Transformer language models are increasingly used as software components, yet biased outputs remain difficult to localize and repair inside the model. Existing fairness testing and repair methods largely operate at the input-output or retraining level, while recent work suggests that bias-related behavior can concentrate in a small set of attention heads.

Summary generated by The Flow from the publisher's feed. The full article lives at Hugging Face Trending Papers.

arXiv Machine Learning

Jul 15

Toward Localizing and Repairing Bias in Transformer Attention Heads

arXiv:2607. 12863v1 Announce Type: cross Abstract: Transformer language models are increasingly used as software components, yet biased outputs remain difficult to localize and repair inside the model.

By Sigma Jahan

llms safety

arXiv Machine Learning

Jun 29

Prism Transformer: Progressive Head Schedules for Hierarchical Attention Processing

arXiv:2606. 27449v1 Announce Type: new Abstract: Multi-head attention conventionally partitions the hidden dimension equally across all heads at every layer, enforcing an identical representational subspace dimension (dh = dmodel/h) throughout the models depth.

By Shubham Aggarwal

llms benchmarks

arXiv AI

Jun 17

Top-Theta Attention: Sparsifying Transformers by Compensated Thresholding

arXiv:2502. 08363v3 Announce Type: replace-cross Abstract: We present Top-Theta (Top-$\theta$) Attention, a training-free method for sparsifying transformer attention during inference.

By Konstantin Berestizshevsky, Renzo Andri, Lukas Cavigelli

llms nlp efficiency

arXiv Machine Learning

Jun 8

Attention Sink in Transformers: A Survey on Utilization, Interpretation, and Mitigation

arXiv:2604. 10098v2 Announce Type: replace Abstract: As the foundational architecture of modern machine learning, Transformers have driven remarkable progress across diverse AI domains.

By Zunhai Su, Hengyuan Zhang, Wei Wu, Yifan Zhang, Yaxiu Liu, He Xiao, Qingyao Yang, Yuxuan Sun, Rui Yang, Chao Zhang, Jing Xiong, Hui Shen, Keyu Fan, Weihao Ye, Chaofan Tao, Taiqiang Wu, Zhongwei Wan, Tiantian Zhang, Bowen Yan, Zhen Li, Yiming Zhang, Congkai Xie, Yulei Qian, Yuchen Xie, Yik-Chung Wu, Hongxia Yang, Ngai Wong

llms safety

arXiv AI

Jul 10

Adaptive Generation of Bias-Eliciting Questions for LLMs

arXiv:2510. 12857v2 Announce Type: replace-cross Abstract: Large language models (LLMs) are now widely deployed in user-facing applications, reaching hundreds of millions of users worldwide.

By Robin Staab, Jasper Dekoninck, Maximilian Baader, Martin Vechev

llms benchmarks safety

arXiv Machine Learning

Jul 31

Fairness Pruning: Locating Demographic Bias in GLU-MLP Layers via Differential Activations

arXiv:2607. 28319v1 Announce Type: cross Abstract: This work presents Fairness Pruning, a lightweight structural intervention method designed for the management and future mitigation of demographic bias in large language models (LLMs).

By Pere Martra, Eugenio Mart\'inez C\'amara, Alfonso Ure\~na L\'opez

llms efficiency benchmarks safety

Related stories

Toward Localizing and Repairing Bias in Transformer Attention Heads

Prism Transformer: Progressive Head Schedules for Hierarchical Attention Processing

Top-Theta Attention: Sparsifying Transformers by Compensated Thresholding

Attention Sink in Transformers: A Survey on Utilization, Interpretation, and Mitigation

Adaptive Generation of Bias-Eliciting Questions for LLMs

Fairness Pruning: Locating Demographic Bias in GLU-MLP Layers via Differential Activations