arXiv Machine Learning By Arkanath Pathak, Unnat Jain, Alexander C. Berg

Repurposing Pre-trained LLMs as High Fidelity Continuous Text Autoencoders

Read the original on arXiv Machine Learning →

The paper introduces LLMAE, a technique that transforms a pretrained decoder-only language model into a continuous text autoencoder by inserting a fixed-length latent bottleneck into its internal activations. Using a 270M Gemma 3 model with structured attention masks, LoRA adaptation, and KL regularization, LLMAE achieves near-perfect reconstruction of text sequences up to 1024 tokens. The authors further show that the resulting latent representation can be leveraged to train a latent text diffusion model for detailed image captioning, demonstrating downstream utility.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

arXiv Computer Vision
6d ago

RepFusion: Leveraging Multimodal Priors for Denoising in Representation Space

RepFusion leverages multimodal large language models (MLLMs) as noisy representation encoders to condition a diffusion transformer for denoising in representation space. By reusing MLLM outputs across denoising steps, the method achieves faster inference while maintaining performance comparable to per-step recomputation. Experiments show RepFusion outperforms baselines with similar inference budgets, demonstrating that MLLMs provide strong priors for denoising visual representations.

By Xichen Pan, Aashu Singh, Satya Narayan Shukla, Xiangjun Fan, Shlok Kumar Mishra, Saining Xie
arXiv Computation and Language
Aug 28

Visual Information-Guided Parallel Decoding for Diffusion Multimodal Large Language Models

Visual Information-Guided Parallel Decoding for Diffusion Multimodal Large Language Models introduces the VIG‑Sampler, a method that prioritizes tokens for decoding based on their attention to image tokens and penalizes redundancy in image‑attention distributions. The approach aims to improve the quality of multimodal generation by selecting more informative tokens during diffusion decoding. Experiments on seven captioning and VQA benchmarks with three open‑source dMLLMs show that VIG‑Sampler outperforms the Info‑Gain Sampler by an average of 19.3 CIDEr points and achieves better COCO Caption results using only half as many decoding steps.

By Insu Lee, Wooje Park, Wonseok Shin, Jinwoo Son, Byonghyo Shim