arXiv Machine Learning

Repurposing Pre-trained LLMs as High Fidelity Continuous Text Autoencoders

The paper introduces LLMAE, a technique that transforms a pretrained decoder-only language model into a continuous text autoencoder by inserting a fixed-length latent bottleneck into its internal activations. Using a 270M Gemma 3 model with structured attention masks, LoRA adaptation, and KL regularization, LLMAE achieves near-perfect reconstruction of text sequences up to 1024 tokens. The authors further show that the resulting latent representation can be leveraged to train a latent text diffusion model for detailed image captioning, demonstrating downstream utility.

arXiv Computer Vision
6d ago

RepFusion: Leveraging Multimodal Priors for Denoising in Representation Space

RepFusion leverages multimodal large language models (MLLMs) as noisy representation encoders to condition a diffusion transformer for denoising in representation space. By reusing MLLM outputs across denoising steps, the method achieves faster inference while maintaining performance comparable to per-step recomputation. Experiments show RepFusion outperforms baselines with similar inference budgets, demonstrating that MLLMs provide strong priors for denoising visual representations.

By Xichen Pan, Aashu Singh, Satya Narayan Shukla, Xiangjun Fan, Shlok Kumar Mishra, Saining Xie
arXiv Computation and Language
Aug 28

Visual Information-Guided Parallel Decoding for Diffusion Multimodal Large Language Models

Visual Information-Guided Parallel Decoding for Diffusion Multimodal Large Language Models introduces the VIG‑Sampler, a method that prioritizes tokens for decoding based on their attention to image tokens and penalizes redundancy in image‑attention distributions. The approach aims to improve the quality of multimodal generation by selecting more informative tokens during diffusion decoding. Experiments on seven captioning and VQA benchmarks with three open‑source dMLLMs show that VIG‑Sampler outperforms the Info‑Gain Sampler by an average of 19.3 CIDEr points and achieves better COCO Caption results using only half as many decoding steps.

By Insu Lee, Wooje Park, Wonseok Shin, Jinwoo Son, Byonghyo Shim
arXiv Computer Vision
Sep 14

Unified Text-Image Generation with Weakness-Targeted Post-Training

The paper introduces a post‑training approach that enables a single inference process to transition from text reasoning to image synthesis, eliminating the need for explicit modality switching. Using the 14B BAGEL model, the authors demonstrate that targeted post‑training data and reward‑weighted training improve multimodal image generation across four independent T2I benchmarks. The study highlights the benefits of joint text‑image generation and strategic data selection for enhancing T2I performance.

By Jiahui Chen, Philippe Hansen-Estruch, Xiaochuang Han, Yushi Hu, Emily Dinan, Amita Kamath, Michal Drozdzal, Reyhane Askari-Hemmat, Luke Zettlemoyer, Marjan Ghazvininejad
arXiv Computer Vision
Sep 16

FLAT: Resampling Image and Text into 1D Flexible-Length Aligned Transmodal Tokens for Retrieval and Generation

FLAT (Flexible‑Length Aligned Transmodal representations) is a joint multimodal pre‑training framework that learns a shared encoder for images and text, producing 1‑D continuous embeddings that can be directly used by downstream generative decoders. By combining contrastive alignment with bidirectional cross‑modal generative objectives, FLAT yields representations that are both discriminative and generative, enabling cross‑modal retrieval and generation with a single pre‑training stage. The model achieves strong performance on T2I generation (GenEval 71.1), image captioning (BLEU‑4 40.5, CIDEr 138.6), and retrieval tasks (Recall@5 86.8/75.8 on MS‑COCO, 98.3/93.6 on Flickr30K), and supports linear interpolation, latent space arithmetic, and zero‑shot composed retrieval.

By Guangyu Sun, Shlok Kumar Mishra, Wentao Bao, Robert Zhenheng Yang, Xiao Wang, Xiyuan Wang, Yujunrong Ma, Chen Yuan, Max Xiangjun Fan, Jun Xiao, Jianpeng Cheng
arXiv Computer Vision
Sep 18

Understanding and Exploiting Diagonal Attention Sparsity in Autoregressive Image Generation

The paper investigates how attention sparsity behaves in autoregressive image generation, finding a distinct diagonal sparsity pattern due to spatial locality of visual tokens. It introduces a diagonal‑aware sparse attention mechanism that skips KV entries along the diagonal within a recent window, achieving up to 3.1× higher throughput and 1.19× lower latency with less than 2% quality loss compared to dense inference.

By Daeun Kim, Junwha Hong, Changhun Oh, Yoonsung Kim, Yoonhyeong Lee, Jongse Park
arXiv Computation and Language
Oct 23

Cola2: Co-Adaptive Hierarchical Language Modeling with Continuous Semantic Compression and Balanced Discrete Generation

Cola2 is a hierarchical language modeling framework that jointly optimizes continuous semantic compression, reconstruction–generation balance, and continuous–discrete co‑adaptation. It dynamically segments text into semantic spans, controls compression via negative feedback, and trains a block‑causal masked diffusion decoder with clean, noisy, and dropped latent conditions. Experiments on five downstream tasks with 210 billion training tokens show competitive scaling, and an unconverged checkpoint with 346 billion source tokens outperforms Llama‑3‑8B on HellaSwag and StoryCloze.

By Hongcan Guo, Yian Zhao, Feng Wang, Shen Nie, Guoqiang Wei, Yan Zeng, Hengshuang Zhao
arXiv AI
Nov 5

MONET: A Massive, Open, Non-redundant and Enriched Text-to-image dataset

MONET is an open, Apache 2.0-licensed dataset comprising 103.8 million curated image‑text pairs, sourced from 2.9 billion raw pairs. The dataset undergoes extensive safety filtering, domain filtering, duplicate removal, and re‑captioning with multiple vision‑language models, and is further enriched with synthetic samples. Each image includes pre‑computed embeddings and annotations to facilitate downstream research, and training a 4B‑parameter latent diffusion model on MONET yields competitive GenEval and DPG scores, proving its utility for large‑scale, reproducible text‑to‑image studies.

By Benjamin Aubin, Gonzalo I\~naki Quintana, Onur Tasar, Sanjeev Sreetharan, Urszula Czerwinska, Damien Henry, Cl\'ement Chadebec