Transformer-based Encoder-Decoder Models

T5Gemma: A new collection of encoder-decoder Gemma models

Introducing T5Gemma, a new collection of encoder-decoder LLMs.

On the quantitative analysis of decoder-based generative models

How to generate text: using different decoding methods for language generation with Transformers

Leveraging Pre-trained Language Model Checkpoints for Encoder-Decoder Models

Geometry-Preserving Encoder/Decoder in Latent Generative Models

arXiv:2501. 09876v3 Announce Type: replace-cross Abstract: Generative modeling aims to generate new data samples that resemble a given dataset.

Understanding Rate-Distortion Performance in Distributed Transformer Inference

arXiv:2601. 22002v5 Announce Type: replace Abstract: Transformers achieve superior performance on many tasks, but impose heavy compute and memory requirements during inference.

Ettin Suite: SoTA Paired Encoders and Decoders

Universal Assisted Generation: Faster Decoding with Any Assistant Model

Performance-Efficiency Tradeoffs in Transformers: An Approximation Theory Perspective

The paper examines how to allocate attention heads and head dimensions across Transformer layers to balance expressivity and efficiency. It provides a mathematical analysis of early layers’ role in information extraction and characterizes the trade‑off between head count and dimension under a fixed parameter budget. The authors prove a saturation effect of softmax activations, showing that increasing head dimensions yields diminishing returns, especially for long sequences, and propose strategies for efficient parameter allocation across layers.

Your Transformer Can Hold Two Thoughts at Once: Evidence of Linear Superposition in LLMs

The paper demonstrates that Large Language Models, despite their non‑linear components, exhibit a fundamental linearity property: when inputs from two distinct text streams are linearly combined, the model outputs a superposition of the individual next‑token distributions. This "Superposition Linearity Hypothesis" appears to be an intrinsic feature of the Transformer architecture, tends to weaken during pretraining, but can be largely restored with lightweight fine‑tuning. The authors also present a guided decoding method that separates the superposed outputs, allowing two coherent continuations to be generated from a single forward pass.

Introducing RWKV - An RNN with the advantages of a transformer

Remote VAEs for decoding with Inference Endpoints 🤗