arXiv AI
Jul 8
Transformers converge to invariant algorithmic cores
arXiv:2602. 22600v2 Announce Type: replace-cross Abstract: Training selects for behavior, not circuitry: many weight configurations can implement the same function.
Kan Extension Transformers: A Categorical Unification of Attention, Diffusion, and Predict-Detach Self-Conditioning
arXiv:2605. 27259v2 Announce Type: replace Abstract: We propose Kan Extension Transformers (KETs) as a categorical design language for a diverse group of Transformer implementations.
Related stories
arXiv AI
Jul 23
Geometric Attention: A Regime-Explicit Operator Semantics for Transformer Attention
arXiv:2601. 11618v2 Announce Type: replace-cross Abstract: Geometric Attention (GA) specifies an attention layer by four independent inputs: a finite carrier (what indices are addressable), an evidence-kernel rule (how masked proto-scores and a link induce nonnegative weights), a probe family (which observables are treated as admissible), and an anchor/update rule (which representative kernel is selected and how it is applied).
arXiv Machine Learning
Jul 7
Legible-by-Construction: Attention and End-to-End Transformers
arXiv:2607. 04319v1 Announce Type: cross Abstract: A companion paper showed that a transformer's feed-forward layer can be rebuilt from explicit fuzzy set operations - intersection, set-difference, and a self-forgetting sequence quantifier - so its hidden units read as named logical operators at no cost to language-model quality.
arXiv AI
Jun 2
DLLM-JEPA: Joint Embedding Predictive Architectures for Masked Diffusion Language Models
arXiv:2606. 00091v1 Announce Type: cross Abstract: Joint Embedding Predictive Architectures (JEPAs) have reshaped self-supervised representation learning in vision.
arXiv Machine Learning
Aug 4
TextNCA: Neural Cellular Automata for Language Modeling via Hierarchical Local Attention
arXiv:2608. 02050v1 Announce Type: cross Abstract: Can a strictly local, iterated, weight-shared computation primitive support language modelling, and which of those three properties actually drives the model's behaviour?
Hugging Face Trending Papers
Jul 5
Legible-by-Construction: Attention and End-to-End Transformers
A companion paper showed that a transformer's feed-forward layer can be rebuilt from explicit fuzzy set operations - intersection, set-difference, and a self-forgetting sequence quantifier - so its hidden units read as named logical operators at no cost to language-model quality. That left the other half of the transformer opaque.
arXiv AI
Jun 8
Data-Efficient Autoregressive-to-Diffusion Language Models via On-Policy Distillation
arXiv:2606. 06712v1 Announce Type: cross Abstract: We study the transformation of autoregressive models (ARLMs) into diffusion language models (DLMs).
arXiv Machine Learning
Jul 30
A Compositional Theory of Causally Masked Transformers
arXiv:2607. 26988v1 Announce Type: cross Abstract: What types of decision problems can a causally masked, finite-precision transformer solve for inputs of arbitrary length?
arXiv AI
Aug 5
A Theory of Conditional Collapse under Low-Rank Weight-Space Ablations: I. The Single-Block Theory and Synthetic Validation
arXiv:2608. 03620v1 Announce Type: cross Abstract: Activation patching and weight-space ablation both claim a component is causally responsible for a behavior, yet they act on different objects: one forward pass versus the parameters behind every forward pass.
arXiv AI
5d ago
Architecture Before the Formula: Individuating Neural Architecture Beyond the Composite Map
arXiv:2601. 11618v3 Announce Type: replace-cross Abstract: Neural architecture is often identified by module syntax, computation graphs, or the composite functions they realize.
arXiv AI
Jul 20
Induction in Both Directions: A Mechanistic Analysis of In-Context Learning in Masked Diffusion Language Models
arXiv:2607. 15893v1 Announce Type: cross Abstract: While the internal mechanisms of autoregressive (AR) transformers have been studied extensively, much less is known about diffusion language models (DLMs), an emerging alternative that generates text by iterative denoising.
arXiv AI
Jul 21
Every Component is a Lookup: Token Attribution and Composition from a Single Decomposition
arXiv:2605. 23393v2 Announce Type: replace-cross Abstract: Mechanistic interpretability of transformers requires identifying not just which components matter but how they compose into the computational route that produced a prediction.