arXiv Machine Learning September 29, 2026 By Leona Hioki

Complex-Valued Phase-Coherent Transformer

Read the original on arXiv Machine Learning →

The paper introduces the Phase-Coherent Transformer (PCT), a complex-valued architecture that replaces traditional softmax attention with a real-valued, smooth gate applied to L2-normalised query-key similarities. PCT eliminates token competition, preserving phase information across layers, and demonstrates strong generalisation on a variety of mid-scale benchmarks, outperforming both standard softmax Transformers and other complex-valued counterparts. Experiments confirm that the gate design is essential: preserving negatively aligned phase components is crucial for performance, while violating these conditions leads to degradation or collapse on long-range tasks.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

arXiv Machine Learning

Sep 22

Complex-valued Phase-Coherent Transformers

arXiv:2609. 22415v1 Announce Type: new Abstract: Complex-valued Transformers have inherited softmax attention over the raw complex inner product.

By Leona Hioki

llms

arXiv AI

Jul 17

Language as a Wave Phenomenon: Semantic Phase Locking and Interference in Neural Networks

arXiv:2512. 01208v5 Announce Type: replace-cross Abstract: In standard Transformer architectures, semantic importance is often conflated with activation magnitude, obscuring the geometric structure of latent representations.

By Alper Y{\i}ld{\i}r{\i}m, \.Ibrahim Y\"uceda\u{g}

llms

arXiv Computation and Language

Aug 25

Do Value Vectors in Deep Layers Need Context from the Residual Stream?

The paper investigates whether deep transformer layers require context from the residual stream to compute value vectors. It finds that allowing deeper layers to use a context‑free value vector—preserving original token information—significantly improves performance, and adding context afterward yields little extra benefit. The authors introduce Bank of Values (BoV), a lookup table of token‑specific value vectors for the last third of layers, which reduces compute and memory while matching or surpassing prior methods on large models.

By Muyu He, Yuchen Liu, Qingya Huang, Li Zhang

llms benchmarks

arXiv Machine Learning

Aug 10

The Expressive Power of Low Precision Softmax Transformers with (Summarized) Chain-of-Thought

arXiv:2605. 18079v2 Announce Type: replace Abstract: Existing expressivity results for transformers typically rely on hardmax attention, high precision, and other architectural modifications that disconnect them from the models used in practice.

By Moritz Br\"osamle, Stephan Eckstein

llms safety

arXiv Machine Learning

Sep 1

Tracing distinguishability through transformer processing with stochastic LayerNorm

arXiv:2608.30720v1 Announce Type: new Abstract: Representational similarity is foundational to analyses of deep networks, yet distances between point-valued representations are not intrinsically tied...

By Kieran Murphy

llms fine-tuning efficiency safety

arXiv Machine Learning

Jun 5

Exact Linear Attention

arXiv:2605. 18848v3 Announce Type: replace Abstract: This paper introduces Exact Linear Attention (ELA), a mechanism that achieves linear computational complexity for Transformer attention by exploiting the exact decomposition property of kernel functions, thereby eliminating approximation error.

By Weinuo Ou

llms computer-vision reinforcement-learning efficiency safety

One email a morning, machine-written

Related stories

Complex-valued Phase-Coherent Transformers

Language as a Wave Phenomenon: Semantic Phase Locking and Interference in Neural Networks

Do Value Vectors in Deep Layers Need Context from the Residual Stream?

The Expressive Power of Low Precision Softmax Transformers with (Summarized) Chain-of-Thought

Tracing distinguishability through transformer processing with stochastic LayerNorm

Exact Linear Attention