Before Q, K, and V: Reconstructing the Transformer

Toward Manifest Relationality in Transformers via Symmetry Reduction

arXiv:2602. 18948v2 Announce Type: replace Abstract: Transformer models contain substantial internal redundancy arising from coordinate-dependent representations and continuous symmetries, in model space and in head space, respectively.

Do Transformers Need Three Projections? Systematic Study of QKV Variants

arXiv:2606. 04032v1 Announce Type: cross Abstract: Transformers have become the standard solution for various AI tasks, with the query, key, and value (QKV) attention formulation playing a central role.

Train your first Decision Transformer

What Is the Minimum Architecture for Prolepsis? Early Irrevocable Commitment Across Tasks in Small Transformers

arXiv:2604. 15010v2 Announce Type: replace-cross Abstract: When do transformers commit to a decision, and what prevents them from correcting it?

A Three-Phase Factual Recall Circuit in Gemma-2B and Gemma-12B-IT

Activation patching reveals how facts are stored, routed, and read out across transformer layers, and why the residual stream does most of the work The post A Three-Phase Factual Recall Circuit in Gemma-2B and Gemma-12B-IT appeared first on Towards Data Science .

Functional Equivalence in Attention: A Comprehensive Study with Applications to Linear Mode Connectivity

arXiv:2606. 17830v1 Announce Type: cross Abstract: Neural network parameter spaces are inherently non-injective, as distinct parameter configurations can realize identical functions through functional equivalence.

How we sped up transformer inference 100x for 🤗 API customers

Cross-Layer Subspace Coupling for LLM Compression: A Unifying Framework and Its Empirical Limits

arXiv:2605. 30836v2 Announce Type: replace Abstract: Recent SVD based compression methods for large language models like SVD LLM and Basis Sharing can be unified under one optimization problem.

Where does Absolute Position come from in decoder-only Transformers?

arXiv:2606. 06160v1 Announce Type: new Abstract: RoPE-trained transformers distinguish absolute position in their attention patterns, even though RoPE encodes only relative offsets in the inner product.

Transformer Geometry Observatory TGO-IV: Developmental Topology Observatory

arXiv:2608. 09997v1 Announce Type: new Abstract: Transformers have had a profound impact on the world of language processing and computer vision.

On the Expressive Power of Transformers

arXiv:2608. 12671v1 Announce Type: new Abstract: Multi-layer transformers form the critical component of essentially all large language models (LLMs) in use today.

Attention Sink in Transformers: A Survey on Utilization, Interpretation, and Mitigation

arXiv:2604. 10098v2 Announce Type: replace Abstract: As the foundational architecture of modern machine learning, Transformers have driven remarkable progress across diverse AI domains.