The paper proposes a self‑supervised representation learning method that serves as a tokenizer for generative models. It supervises a representation loss by factorizing pointwise mutual information, estimated through differences between conditional predictions of a learned generative model. The authors demonstrate that this approach avoids dimensional collapse and feature suppression seen in methods like DINO and LeJEPA, scales predictably with compute, and achieves compute‑optimal performance over direct diffusion for pixel‑space generation, with strong scaling laws on ImageNet and GPIC.
arXiv:2505. 18227v4 Announce Type: replace-cross Abstract: In Transformer architectures, tokens\textemdash discrete units derived from raw data\textemdash are formed by segmenting inputs into fixed-length chunks.
arXiv:2609.36348v1 Announce Type: cross Abstract: Generative and representation learning remain asymmetrically connected: semantic representations are used to improve diffusion generation, whereas th...
FeaTok introduces feature-loss tokenization for image generation, training discrete tokens by reconstructing pretrained features instead of pixels. This approach improves generative performance across multiple self-supervised models (SigLIP, SigLIP2, DINOv2, DINOv3). In class-conditional generation, FeaTok reduces gFDr6 by up to 29% at 256×256 resolution and outperforms the strongest unguided continuous baseline with 38× less sampling compute, while also achieving higher human-preference reward-model scores and surpassing text-conditional baselines after only 500 fine-tuning updates.
Discrete image generators learn the distribution of tokens produced by an image tokenizer. These token representations should be easy for generators to model, not merely enable faithful image reconstr...
arXiv:2607. 27372v1 Announce Type: new Abstract: The deep learning revolution, kicked off by AlexNet, taught us that end-to-end training beats decomposing a problem into hand-designed stages.
arXiv:2609.39222v1 Announce Type: new Abstract: High-compression tokenizers are essential for scaling latent image generative models. However, aggressive compression creates a fundamental tradeoff be...
The paper introduces SCION, a single-model framework that jointly learns generative and self‑supervised representation tasks without relying on labels or pretrained encoders. SCION uses a pixel‑space encoder conditioned on a flow timestep and a shared embedding, training contrastive and flow‑matching objectives simultaneously with gradient‑norm balancing and stop‑gradient techniques. On ImageNet 256×256, SCION achieves state‑of‑the‑art FID scores (8.92 with JiT‑B and 5.89/3.47 with JiT‑L) surpassing prior class‑unconditional and pretrained‑conditioned methods.
The paper introduces MTAR, a training framework for autoregressive image generation that enhances performance through multi-token prediction, token-level contrastive regularization, and semantic dropping. These components address sparse supervision, improve representation discriminability, and accelerate training without affecting inference. On ImageNet, MTAR outperforms LlamaGen with lower FID and faster training, achieving comparable results in only a third of the iterations.
arXiv:2606. 05552v1 Announce Type: new Abstract: Despite progress in image tokenization, standard methods encode redundant information by mixing all granularities within each token, thus redundancy persists between tokens.
arXiv:2609.24691v1 Announce Type: new Abstract: Latent diffusion models now dominate medical image generation, and every such pipeline rests on a \emph{tokenizer} that compresses images into the late...
arXiv:2608.23864v1 Announce Type: new Abstract: Visual tokenizers increasingly inject semantic supervision into latent spaces to make downstream diffusion easier. Yet how these semantics should be or...
arXiv:2603.08064v3 Announce Type: replace Abstract: Most evaluations of generative models rely on feature-distribution metrics such as FID, which operate on continuous recognition features that are e...