arXiv Machine Learning June 17, 2026 By Aryan Sharma, Cutter Dawes, Shivam Raval

Dissociating Decodability and Causal Use in Bracket-Sequence Transformers

Read the original on arXiv Machine Learning →

arXiv:2604. 22128v2 Announce Type: replace-cross Abstract: When trained on tasks requiring an understanding of hierarchical structure, transformers have been found to represent this hierarchy in distinct ways: in the geometry of the residual stream, and in stack-like attention patterns maintaining a last-in, first-out ordering.

Summary generated by The Flow from the publisher's feed. The full article lives at arXiv Machine Learning.

llms

arXiv AI

Jun 3

Causal Evidence of Stack Representations in Modeling Counter Languages Using Transformers

arXiv:2606. 03398v1 Announce Type: cross Abstract: Formal languages have proven to be effective conduits to understand the inner mechanisms of transformers.

By Nishit Singh

llms benchmarks

arXiv AI

Jul 20

Induction in Both Directions: A Mechanistic Analysis of In-Context Learning in Masked Diffusion Language Models

arXiv:2607. 15893v1 Announce Type: cross Abstract: While the internal mechanisms of autoregressive (AR) transformers have been studied extensively, much less is known about diffusion language models (DLMs), an emerging alternative that generates text by iterative denoising.

By Andy Catruna, Emilian Radoi

llms rag diffusion

arXiv Machine Learning

Jun 17

An expressivity analysis of hierarchical modelling in deep transformers via bounded-depth grammars

arXiv:2606. 17522v1 Announce Type: cross Abstract: Deep neural networks are widely believed to derive their expressive power from their ability to form \textbf{hierarchical representations}, capturing progressively more abstract and compositional features across layers.

By Vinoth Nandakumar, Qiang Qu, Pramod Thebe, Sakshi Khachariya, Tongliang Liu

llms

arXiv AI

Jul 14

Disentangling Feature Structure: A Mathematically Provable Two-Stage Training Dynamics in Transformers

arXiv:2502. 20681v3 Announce Type: replace-cross Abstract: Transformers may exhibit two-stage training dynamics during the real-world training process.

By Zixuan Gong, Shijia Li, Yong Liu, Jiaye Teng

llms

arXiv AI

Aug 7

Beyond Sequence Order: Syntax-Informed Positional Embeddings for Transformers

arXiv:2608. 06111v1 Announce Type: cross Abstract: Positional embeddings (PE) in Transformers encode token distance and order but are largely agnostic to \textit{syntactic structure}.

By Haris Riaz, Hyungji Kim, Mihai Surdeanu

llms rag benchmarks

arXiv Machine Learning

Jul 30

Dense Local Dependencies Induce Attention-Logit Explosion and Training Instability During Long-Sequence Transformer Training

arXiv:2505. 15548v2 Announce Type: replace Abstract: Autoregressive transformer language models frequently exhibit training instability when trained on long sequences, particularly under low-precision arithmetic.

By Suvadeep Hajra

llms

Related stories

Causal Evidence of Stack Representations in Modeling Counter Languages Using Transformers

Induction in Both Directions: A Mechanistic Analysis of In-Context Learning in Masked Diffusion Language Models

An expressivity analysis of hierarchical modelling in deep transformers via bounded-depth grammars

Disentangling Feature Structure: A Mathematically Provable Two-Stage Training Dynamics in Transformers

Beyond Sequence Order: Syntax-Informed Positional Embeddings for Transformers

Dense Local Dependencies Induce Attention-Logit Explosion and Training Instability During Long-Sequence Transformer Training