arXiv:2609.40362v1 Announce Type: new Abstract: We present Multimodal Flow, a fully continuous generative model of language and vision. Most unified multimodal models either model both language and q...
Multimodal models often build on architectures designed for generative vision-language modeling, typically combining separately pretrained vision encoders with causal language models. Visual document...
NeoMME is a family of 260M and 800M‑parameter multimodal‑native multilingual encoders that process text and raw image patches in a single bidirectional Transformer. Trained from scratch with a masked discrete‑diffusion objective conditioned on visible image patches, NeoMME supports a 16,384‑token context, enabling encoding of up to two 4K UHD images. In downstream tests, NeoMME‑Retriever models outperform all sub‑800M‑parameter baselines on the ViDoRe v3 benchmark and achieve twice the throughput of ColModernVBERT on an NVIDIA L40S, while hierarchical token pooling and asymmetric quantization compress embeddings 255× with minimal loss in retrieval performance.
The paper introduces SCION, a single-model framework that jointly learns generative and self‑supervised representation tasks without relying on labels or pretrained encoders. SCION uses a pixel‑space encoder conditioned on a flow timestep and a shared embedding, training contrastive and flow‑matching objectives simultaneously with gradient‑norm balancing and stop‑gradient techniques. On ImageNet 256×256, SCION achieves state‑of‑the‑art FID scores (8.92 with JiT‑B and 5.89/3.47 with JiT‑L) surpassing prior class‑unconditional and pretrained‑conditioned methods.
Traditional multimodal representation learning and generation are two stages: a contrastive or self-supervised visual encoder is trained first, followed by a separate downstream generative model. This...
The paper investigates how new concepts can be integrated into unified multimodal models (UMMs) by separating generation and understanding objectives through a novel visual entity bound to a single task direction. Experiments show that the effectiveness of cross‑task usability depends on where the concept is injected into the shared computation, with a mid‑stack alignment objective achieving high concept acquisition with minimal loss to overall performance. The study highlights that unified weights alone are insufficient; the two directions must share a semantic format at the entry point for efficient concept integration.
arXiv:2607. 04423v1 Announce Type: cross Abstract: Unified Multimodal Models (UMMs) integrate image understanding and generation within a single architecture, yet how the two tasks interact remains understudied.
Vision-language models commonly project all tokens produced by a pretrained vision encoder into a large language model. However, final-layer features can discard text, local attributes, and spatial relationships, while high-resolution inputs substantially increase context length and inference latency.
arXiv:2509. 07295v4 Announce Type: replace-cross Abstract: Unified multimodal models (UMMs) unify visual understanding and generation within a single architecture.
arXiv:2608. 05000v1 Announce Type: cross Abstract: Vision offers a critical axis for advancing foundation models, driving a shift towards natively unified multimodal pretraining.
The paper investigates image tokenizers as the visual language of unified multimodal models by creating a controlled autoregressive testbed that tracks task‑specific validation losses during multimodal continual pretraining across text, image, text‑to‑image, and image‑to‑text predictions. It shows that losses must be analyzed by task, that the loss–performance relationship varies with the token space, and that better reconstruction does not always lead to stronger downstream performance. The study also demonstrates how tokenizer design choices—such as discriminator use, semantic supervision, and vocabulary size—affect joint modeling and downstream results.
FLAT (Flexible‑Length Aligned Transmodal representations) is a joint multimodal pre‑training framework that learns a shared encoder for images and text, producing 1‑D continuous embeddings that can be directly used by downstream generative decoders. By combining contrastive alignment with bidirectional cross‑modal generative objectives, FLAT yields representations that are both discriminative and generative, enabling cross‑modal retrieval and generation with a single pre‑training stage. The model achieves strong performance on T2I generation (GenEval 71.1), image captioning (BLEU‑4 40.5, CIDEr 138.6), and retrieval tasks (Recall@5 86.8/75.8 on MS‑COCO, 98.3/93.6 on Flickr30K), and supports linear interpolation, latent space arithmetic, and zero‑shot composed retrieval.