arXiv Computer Vision

FuseReg: Regularizing Layer Fusion Mitigates the Reconstruction-Generation Gap in Representation Autoencoders

arXiv Machine Learning
Oct 12

Decoding Register Tokens Preserves Global Scene Details

The paper investigates whether keeping non‑spatial register tokens in vision transformer decoders improves image reconstruction and generation. Using representation autoencoders with frozen register‑equipped encoders, the authors show that joint self‑attention over both patch and register tokens yields better reconstruction quality on ImageNet‑1K, with lower FID scores and higher PSNRs compared to patch‑only baselines. For generation, a multi‑head attention pooling predictor recovers register features from sampled patches, further reducing FID by nearly 10% over the patch‑only approach.

By David Jacob Drexlin, Julius Hense, Jonas Dippel
arXiv Computer Vision
Oct 28

WithinVAE: Compact Visual Autoencoding with Self-Distilled Semantics

WithinVAE is a compact variational visual autoencoder that learns semantic structure internally using a sequential bottleneck and self‑distillation from an exponential‑moving‑average copy of itself. It achieves a four‑fold compression ratio on ImageNet‑1K, improving reconstruction metrics such as PSNR by 1.36 dB over RPiAE while using half the latent capacity. The model also boosts linear probing performance, demonstrating that semantic gains arise from self‑distillation rather than external supervision.

By Yuzhou Ji, Jiepeng Wang, Tianyu Jin, Lizhuang Ma, Haibin Huang, Chi Zhang, Xuelong Li
arXiv AI
Oct 19

Rethinking the design space of Vanilla Pixel Diffusion Transformers

The paper revisits pixel-space diffusion models, showing that a vanilla diffusion transformer can generate directly in pixel space by swapping velocity prediction for clean image prediction. It introduces MFlow, an adaptive parameterization that interpolates between these two prediction modes based on data representation and model capacity, and demonstrates competitive performance on ImageNet, UCF‑101, and text‑to‑image tasks. Additionally, the authors add ConvRefiner, a lightweight convolutional module trained post‑hoc to restore high‑frequency details while preserving FID.

By Soumik Mukhopadhyay, Long Mai, Abhinav Shrivastava, Aniruddha Mahapatra
arXiv Machine Learning
Oct 9

One Block, Multiple Depths: Recurrent Vision Transformers with Depth-Programmed Experts

The paper introduces reViT, a recurrent vision transformer that uses a single Transformer block repeatedly to achieve the accuracy of a full-depth encoder while keeping inference FLOPs comparable. It achieves depth‑specific transformations by expressing each block’s feed‑forward network as a convex combination of a small shared expert bank, with a continuous depth coordinate guiding the mixture. Experiments on ImageNet‑1k and DINOv2 distillation show that reViT matches or surpasses state‑of‑the‑art models, reduces stored parameters by about 70%, and supports elastic‑depth deployment from a single checkpoint.

By Adrian Bulat, Yassine Ouali, Georgios Tzimiropoulos
Hugging Face Trending Papers
Aug 13

V-RAE: Rethinking Video Latent Spaces for Generation

Latent video generation relies on autoencoders to define a compact space in which generative models operate. Although video autoencoder architectures have evolved substantially, their latent spaces are still optimized primarily for pixel-level reconstruction and provide limited high-level semantic organization.