arXiv AI
Jun 26
Reconstruction Alignment Improves Unified Multimodal Models
arXiv:2509. 07295v4 Announce Type: replace-cross Abstract: Unified multimodal models (UMMs) unify visual understanding and generation within a single architecture.
Improved Baselines with Representation Autoencoders
arXiv:2605. 18324v2 Announce Type: replace-cross Abstract: Representation Autoencoders (RAE) replace traditional VAE with pretrained vision encoders.
Related stories
arXiv Machine Learning
Jul 31
Explorative Modeling: Unlocking a Third Pretraining Axis and End-to-End Generation
arXiv:2607. 27372v1 Announce Type: new Abstract: The deep learning revolution, kicked off by AlexNet, taught us that end-to-end training beats decomposing a problem into hand-designed stages.
arXiv AI
23h ago
Where a New Concept Must Enter: Entry Point Gates Cross-Task Usability in Unified Multimodal Models
arXiv:2608. 17564v1 Announce Type: cross Abstract: Unified multimodal models (UMMs) are motivated by the hope that understanding and generation reinforce each other but controlled ablations repeatedly find that adding a generation objective leaves understanding flat.
arXiv Machine Learning
Aug 4
UDT: Reconciling U-Nets and Diffusion Transformers with Data-Adaptive Token Reduction
arXiv:2608. 01298v1 Announce Type: cross Abstract: Diffusion Transformers (DiTs) have emerged as a core architecture in generative modeling due to their scalability and adaptability to multimodal tasks.
arXiv AI
Aug 11
LF${}^{2}$AR: Accounting for Layerwise Dynamics to Improve Multimodal Adaptation of Language Models
arXiv:2503. 06211v3 Announce Type: replace-cross Abstract: Text-pretrained language models (LMs) encode rich world knowledge, but adapting them to process and generate perceptual modalities such as audio and images while effectively leveraging that knowledge remains challenging.
arXiv Machine Learning
Jun 2
Reconsidering Positional Supervision in Masked Diffusion Language Model Training
arXiv:2601. 22947v2 Announce Type: replace-cross Abstract: Masked diffusion language models (MDLMs) generate text by unmasking tokens in parallel and have recently emerged as alternatives to autoregressive language models.
arXiv AI
Jun 15
Learning What to Predict: Downstream-Guided Task Design for Continued Pretraining
arXiv:2601. 22108v2 Announce Type: replace-cross Abstract: Continued pretraining is optimized with fixed self-supervised tasks but selected by downstream performance, creating a coarse feedback loop in which practitioners evaluate checkpoints, change data mixtures or objectives, and restart runs, while individual updates remain blind to target capabilities.
Hugging Face Trending Papers
Jul 2
DRDN: Decoupled Representation Dynamic Network for From-Scratch ViT Class-Incremental Learning
Dynamic expansion methods for class-incremental learning (CIL) protect task-specific knowledge by growing dedicated tokens or subnetworks, yet our analyses suggest that classification supervision alone does not sufficiently preserve task-agnostic shared backbone representations over long incremental sequences. We identify two intertwined challenges: cross-task confusion from sequential training on predominantly current-task data, which biases decision boundaries toward recent tasks; and under-optimized shared representations in the backbone that cap long-term discriminability as tasks accumulate.
arXiv AI
Jun 17
Rethinking Cross-Layer Information Routing in Diffusion Transformers
arXiv:2605. 20708v2 Announce Type: replace-cross Abstract: Diffusion Transformers (DiTs) have become a de facto backbone of modern visual generation, and nearly every major axis of their design -- tokenization, attention, conditioning, objectives, and latent autoencoders -- has been extensively revisited.
arXiv AI
Jul 31
Decoupled Visual Processing: Efficient Multimodal Adaptation via Modality-Specific Transformer Substitution
arXiv:2607. 26596v1 Announce Type: cross Abstract: Multimodal large language models (MLLMs) have demonstrated remarkable capabilities by integrating visual and textual understanding within a unified transformer architecture.
arXiv AI
Jul 22
Norm or Direction? Decoding Vision Mambas for High-Resolution Vision
arXiv:2607. 18625v1 Announce Type: cross Abstract: Vision Mamba models replace quadratic self-attention with linear complexity selective state space models (SSMs), emerging as efficient visual backbones.
arXiv AI
Jul 29
Argus-Unified: Towards A Compact and Economical Unified Model for Image Understanding and Generation
arXiv:2607. 25527v1 Announce Type: cross Abstract: Unifying visual understanding and generation in one model holds immense promise, but remains challenging and expensive due to heavy compute and data demands and conflicts between the visual features needed for these two capabilities.