FeaTok introduces feature-loss tokenization for image generation, training discrete tokens by reconstructing pretrained features instead of pixels. This approach improves generative performance across multiple self-supervised models (SigLIP, SigLIP2, DINOv2, DINOv3). In class-conditional generation, FeaTok reduces gFDr6 by up to 29% at 256×256 resolution and outperforms the strongest unguided continuous baseline with 38× less sampling compute, while also achieving higher human-preference reward-model scores and surpassing text-conditional baselines after only 500 fine-tuning updates.
By Zhang Xiaofeng, Samuel Lavoie, Nicolas Beltran-Velez, Jianhao Yuan, Felix Friedrich, Jakob Verbeek, Aaron Courville, Michal Drozdzal, Adriana Romero-Soriano
The paper proposes a self‑supervised representation learning method that serves as a tokenizer for generative models. It supervises a representation loss by factorizing pointwise mutual information, estimated through differences between conditional predictions of a learned generative model. The authors demonstrate that this approach avoids dimensional collapse and feature suppression seen in methods like DINO and LeJEPA, scales predictably with compute, and achieves compute‑optimal performance over direct diffusion for pixel‑space generation, with strong scaling laws on ImageNet and GPIC.
By Alan Baade, Kyle Sargent, Vincent Sitzmann, Eric Ryan Chan, Justin Johnson, Li Fei-Fei, Gordon Wetzstein
The paper introduces MTAR, a training framework for autoregressive image generation that enhances performance through multi-token prediction, token-level contrastive regularization, and semantic dropping. These components address sparse supervision, improve representation discriminability, and accelerate training without affecting inference. On ImageNet, MTAR outperforms LlamaGen with lower FID and faster training, achieving comparable results in only a third of the iterations.
By Guo Niu, Xiongfei Yao, Teng Wang, Nannan Zhu
QuadTok introduces a quadtree-based visual tokenizer that replaces traditional 2D grids or 1D token sequences with a hierarchical structure, allowing dynamic allocation of tokens to visually complex regions while using coarser resolution for homogeneous areas. The tokenizer, trained on ImageNet, reduces token usage by about 10% compared to a fixed 256-token grid and retains comparable reconstruction fidelity. Leveraging the tree’s natural causality, a 947 M GPT-style model conditioned on a quadtree topology achieves a 2.08 gFID on ImageNet 256×256 and supports zero‑shot spatially controlled image generation.
By Yucheng Mao, Zeyuan Chen, Xiaojun Shan, Xiang Zhang, Divyansh Srivastava, Bingnan Li, Zhuowen Tu
arXiv:2606. 05552v1 Announce Type: new Abstract: Despite progress in image tokenization, standard methods encode redundant information by mixing all granularities within each token, thus redundancy persists between tokens.
By Haozhe Chi, Jinghan Li, Hao Jiang, Wu Sheng, Yi Ma, Jing Wang, Yadong Mu
One-dimensional (1D) variable-length visual tokenizers enable adaptive compression by varying the number of tokens, allowing downstream autoregressive (AR) models to flexibly trade off generation qual...
arXiv:2610.09242v1 Announce Type: new
Abstract: Representation Autoencoders (RAEs) generate images from pre-trained visual fea- tures, but their dense token grids make generative modeling expensive....
By Ram\'on Calvo-Gonz\'alez, Youssef Saied, Fran\c{c}ois Fleuret
arXiv:2609.36756v1 Announce Type: cross
Abstract: One-dimensional (1D) variable-length visual tokenizers enable adaptive compression by varying the number of tokens, allowing downstream autoregressiv...
By Jiawei Zhang, Shuhao Liu, Rong Huang, Yuancheng Li, Zhihui Li, Xiaojun Chang, Changlin Li
arXiv:2407.03463v2 Announce Type: replace-cross
Abstract: In the realm of self-supervised learning (SSL), conventional wisdom has gravitated towards the utility of massive, general domain datasets fo...
By Jes\'us M Rodr\'iguez-de-Vera, Imanol G Estepa, Ignacio Saras\'ua, Bhalaji Nagarajan, Petia Radeva
arXiv:2602.03753v2 Announce Type: replace
Abstract: While representation alignment with self-supervised models has been shown to improve diffusion model training, its potential for enhancing inferenc...
By Nicolas Sereyjol-Garros, Ellington Kirby, Victor Letzelter, Victor Besnier, Nermin Samet
arXiv:2609.39222v1 Announce Type: new
Abstract: High-compression tokenizers are essential for scaling latent image generative models. However, aggressive compression creates a fundamental tradeoff be...
By Xu Huang, Ye Huang, Zijun Liao, Yuwei Niu, Xiaojie Li, Menghan Zhou, De Wen Soh, Xiaotong Li, Daquan Zhou
arXiv:2607. 00371v1 Announce Type: cross Abstract: Visual AutoRegressive modeling (VAR) has pioneered a coarse-to-fine multi-scale autoregressive generative paradigm, demonstrating strong capabilities in image generation.
By Nuoyan Zhou, Zhijun Tu, Lei Yu, Kun Cheng, Jie Hu, Nannan Wang, Xinghao Chen