arXiv AI

Pyramidal Columns Acting as Attention Heads

The paper explores whether a group of pyramidal columns in posterior temporal language regions could serve as a biological counterpart to a Transformer attention head. By mapping the functional organization of attention heads onto neurophysiological mechanisms within and across these columns, the authors propose that such a neuromorphic construction is possible, though empirical confirmation remains open. The study does not insist on reproducing the standard Transformer mathematics, allowing for alternative mechanical and mathematical instantiations consistent with biology.

arXiv AI
Jul 23

Geometric Attention: A Regime-Explicit Operator Semantics for Transformer Attention

arXiv:2601. 11618v2 Announce Type: replace-cross Abstract: Geometric Attention (GA) specifies an attention layer by four independent inputs: a finite carrier (what indices are addressable), an evidence-kernel rule (how masked proto-scores and a link induce nonnegative weights), a probe family (which observables are treated as admissible), and an anchor/update rule (which representative kernel is selected and how it is applied).

By Luis Rosario Freytes
arXiv Machine Learning
Sep 17

Transformation Laws in Neural Representations: Structure, Realisability, and Construction

The paper investigates how neural representations maintain the structure of input changes, linking representation analysis with internal interventions. It characterises when transformations can be applied through an encoder, providing linear settings where defects depend on discarded information and detailing failure modes for rectifiers and harmonic carriers. Using colour as a case study, the authors show that hue orbits in frozen visual features concentrate most energy in the first two harmonics, that this structure is inherited from input and architecture, and that a compact, fixed‑action interface can read hue zero‑shot with low error on unseen shapes.

By Yuan Sun
arXiv Machine Learning
Oct 22

Geometric knowledge spaces, Attention pairing and Architectures

The paper introduces a geometric framework for deriving attention mechanisms and exploring new transformer architectures. It formalizes concepts such as knowledge worlds and attention pairings, showing how this approach naturally yields standard attention, causal attention, linear attention, tensor attention, and recurrent neural networks. The authors also propose several new geometric spaces that could inspire novel neural network designs.

By Shang Xu
arXiv Machine Learning
Sep 4

Modern Transformers Are Implicit Hybrids: From Functional Differentiation to Principled Hybrid Architecture Design

The paper proposes a principled way to design hybrid transformer architectures that combine Full Attention (FA) and Linear Attention (LA). By introducing two intervention metrics—RoPE Frequency Importance Score (RFIS) and RoPE Positional Dependence (RPD)—the authors identify a clear taxonomy of retrieval and positional heads, defining a Global Positional Band (GPBand) that aligns with training-length positional scales. Using these insights, they build a Head‑wise Hybrid Architecture (HwH) that assigns FA to global retrieval and LA to local positional modeling, achieving strong language modeling, improved retrieval, and superior zero‑shot long‑context extrapolation compared to standard Transformers and other hybrids.

By Runlin Shi, Bojian Yin, Guoqi Li
arXiv AI
Sep 30

Which Attention Heads are like the Human Head? Not the Ones that Compute

The study investigates whether attention heads in large language models that align with human EEG signals are causally involved in model computation. By ablating these brain‑aligned heads during a pattern‑completion task, the authors find that while such heads contribute to performance, their removal is less disruptive than removing heads selected by attribution patching. The research also distinguishes two families of brain‑aligned heads—novelty and repetition heads—highlighting that novelty heads track human attention but are less critical than random ablation, whereas repetition heads modestly aid performance and align with abstract‑pattern representations.

By Christopher Pinier, Gustaw Opie{\l}ka, Hannes Rosenbusch, Taylor Webb, Michael D. Nunez, Claire E. Stevenson