Tokenization in Transformers v5: Simpler, Clearer, and More Modular

How to train a new language model from scratch using Transformers and Tokenizers

tokenizers v1: encode, decode and scaling, measured

Distilling Sequential Computation in Transformer Language Models

Transformer language models process sequences token by token in an autoregressive manner, making growing contexts increasingly expensive. Yet many adjacent token spans are highly predictable or freque...

Multiplication Beyond Groups: Stratified Fourier Mechanisms in Transformer Circuits

arXiv:2607. 07066v1 Announce Type: cross Abstract: Transformers have demonstrated a remarkable ability to learn algorithmic reasoning, yet mechanistic analyses have mostly focused on globally invertible operations such as cyclic addition and group composition.

Variable-Length Tokenization via Learnable Global Merging for Diffusion Transformers

arXiv:2606. 20076v1 Announce Type: cross Abstract: Latent Diffusion Models (LDMs) have become dominant in visual synthesis, but their quality-compute trade-off is largely constrained by the tokenizer's fixed compression ratio.

Distilling Sequential Computation in Transformer Language Models

arXiv:2609.27233v1 Announce Type: new Abstract: Transformer language models process sequences token by token in an autoregressive manner, making growing contexts increasingly expensive. Yet many adja...