arXiv AI
Jun 26
Reconstruction Alignment Improves Unified Multimodal Models
arXiv:2509. 07295v4 Announce Type: replace-cross Abstract: Unified multimodal models (UMMs) unify visual understanding and generation within a single architecture.
Making Language the Low-Rate Code of Vision: Successive Refinement from Words to Pixels
Related stories
arXiv Computation and Language
3d ago
PixelBench: Can LLMs Understand and Generate Images Pixel by Pixel?
PixelBench is a benchmark that evaluates large language models on their ability to generate and understand images pixel by pixel, without relying on learned image tokenizers or decoders. The benchmark includes text-to-pixel (T2P) and image-to-pixel (I2P) generation tasks across multiple resolutions, as well as a pixel-understanding evaluation. Models produce an RGB palette followed by a grid of indices that map each pixel to a palette color, and the study shows that strong models can generate recognizable images from text and outperform specialized pixelization algorithms at low resolutions, with performance correlating with pixel-understanding ability and improving up to a point before declining at high resolutions. Additionally, Pixel Reconstruction Tuning (PRT) augments visual instruction tuning with caption‑palette‑grid trajectories, and post‑training Molmo2‑4B with PRT data improves performance on 16 visual‑centric benchmarks, demonstrating the utility of explicit pixel reconstruction as auxiliary supervision.
arXiv Machine Learning
Oct 12
Vision2Code: A Multi-Domain Benchmark for Evaluating Image-to-Code Generation
Vision2Code is a new benchmark for image-to-code generation that covers 15 domains and 539 test examples without relying on reference source code. It evaluates generated programs by executing them and comparing rendered outputs to the original images using domain‑specific rubrics, yielding scores that correlate better with human judgments than generic metrics. The study shows domain variability in model performance and demonstrates that the scoring method can improve fine‑tuning of models like Qwen3.5‑9B and Phi‑4‑MM via rejection sampling.
arXiv Computer Vision
Sep 2
A Glance Is All You Need: Single-Pass Fine-Grained Image Captioning with SimLoss
arXiv:2609.00591v1 Announce Type: new Abstract: An image may be worth a thousand words, but most captioning models describe it in only a few. Modern vision-language models produce fluent high-level c...
arXiv AI
Sep 2
Vision Is Not Overhead: One-Pass Block Drafting for Lossless Speculative Decoding in Vision-Language Models
The paper introduces GLANCE, a one‑pass block drafting method that enables lossless speculative decoding for vision‑language models. By using a block‑diffusion head that reads the fused vision‑language state, GLANCE eliminates the need for the drafter to process the image at every step, allowing it to fill an entire block in a single forward pass. Experiments show that GLANCE can decode up to 2.93× faster than autoregressive decoding while maintaining exact greedy decoding results across multiple tasks.
arXiv AI
Oct 2
LensVLM: Selective Context Expansion for Compressed Visual Representation of Text
LensVLM is an inference framework and post‑training recipe that lets Vision‑Language Models (VLMs) process compressed images of text by selectively expanding only the relevant parts back to full resolution. Using Qwen3.5‑9B‑Base, LensVLM achieves accuracy comparable to full‑text models at 4.3× compression and outperforms other compression baselines up to 10.1× across seven text QA benchmarks, while also improving performance on multimodal document and code tasks as compression increases.
arXiv Computer Vision
5d ago
CompoSET: A Controlled Single-Edit Benchmark for Vision-Language Compositionality
CompoSET is a new vision‑language benchmark that tests compositionality by presenting each image‑caption pair with a foil that differs by only a single controlled edit—such as color, count, spatial relation, or object state—while keeping the overall scene largely unchanged. The dataset contains 1,776 instances across 80 scenes, 16 edit types in four semantic categories, and captions at three verbosity levels, with an average of only 8% pixel change between base and foil images. Evaluation of 22 dual‑encoder VLMs shows that all models perform at or below chance on CompoSET, contrasting with their strong performance on prior benchmarks that conflate global image differences with compositional reasoning.
arXiv AI
Oct 5
VTBench: Evaluating Visual Tokenizers for Autoregressive Image Generation
VTBench is a new benchmark that evaluates visual tokenizers (VTs) used in autoregressive image generation. It assesses VTs on image reconstruction, detail preservation, and text preservation across diverse scenarios, revealing that continuous VAEs outperform discrete VTs in maintaining spatial structure and semantic detail. The study also explores GPT‑4o’s potential autoregressive behavior and releases the benchmark publicly to encourage development of robust, open‑source VTs.
arXiv Computation and Language
Oct 27
Whispers from the Neighbors: Rethinking Caption Consensus as Inference-Time Evidence for Vision-Language Retrieval
arXiv:2610.33798v1 Announce Type: cross Abstract: An image is rarely described only once. Retrieval benchmarks and many collections attach several captions to each picture, yet dual encoders still ju...
arXiv Computer Vision
Oct 1
Rethinking Generative Image Compression at Extremely Low Bitrates
The paper introduces RAE-CoD, a diffusion-based compression method that operates in a representation autoencoder space to preserve recognizable content even at extremely low bitrates. It addresses the problem of semantic collapse observed in existing codecs when the bitrate approaches zero, showing that reconstruction losses conflict with semantic objectives and that VAE diffusion models lose efficiency in preserving semantics. Experiments on MSCOCO-30K demonstrate that RAE-CoD outperforms competitors, reducing VFM feature MSE and Fréchet Distance ratios by at least 25.7% and 69.1% at 0.001–0.008 bpp while maintaining stable recognizability and quality.
arXiv Computation and Language
Aug 28
Visual Information-Guided Parallel Decoding for Diffusion Multimodal Large Language Models
Visual Information-Guided Parallel Decoding for Diffusion Multimodal Large Language Models introduces the VIG‑Sampler, a method that prioritizes tokens for decoding based on their attention to image tokens and penalizes redundancy in image‑attention distributions. The approach aims to improve the quality of multimodal generation by selecting more informative tokens during diffusion decoding. Experiments on seven captioning and VQA benchmarks with three open‑source dMLLMs show that VIG‑Sampler outperforms the Info‑Gain Sampler by an average of 19.3 CIDEr points and achieves better COCO Caption results using only half as many decoding steps.
arXiv Computer Vision
Oct 9
BudgetPix: Compute-Adaptive Tokenization for Pixel-Space Image Diffusion
BudgetPix introduces a compute‑adaptive tokenization framework for pixel‑space image diffusion, dynamically allocating computational resources based on visual complexity and spatial layout. It comprises an adaptive encoder using an entropy‑guided quadtree and multi‑scale patch embedder, a scale‑aware decoder, and a flexible training schedule that allows pixel‑space denoisers to operate across variable token counts. The method integrates with existing diffusion architectures, achieving comparable fidelity to larger models while using only a fraction of the compute budget, and demonstrates improved quality‑efficiency tradeoffs in both text‑to‑image and class‑conditional generation.