RangeSWA is a training‑free method designed to improve length extrapolation in continuous diffusion language models. It combines local sliding‑window attention, remote block summaries, and dynamic content retrieval to maintain local detail while accessing long‑range context. The approach also includes positional remapping, conditioning attention correction, and denoising‑stage‑adaptive window scheduling to mitigate generation degradation, resulting in lower perplexity and fewer repetitions in long‑text generation.
By Guangfu Guo, Xiaoqian Lu, Linsey Pang, Weiran Yao, Haolin Chen, Long Cheng, Kunpeng Liu
The paper introduces a survival-guided length control method for diffusion language models (DLMs), framing length selection as a discrete-time survival problem over the end-of-sequence token. This training‑free, plug‑in length predictor can be added to any existing DLM and reduces unnecessary denoising steps. Experiments on reasoning and code‑generation benchmarks show up to a seven‑fold speedup in inference while maintaining task accuracy, and reveal that predicted lengths vary significantly even within the same dataset, affecting model performance.
By Ivan Kobyzev, Abbas Ghaddar, Yufei Cui
arXiv:2606. 08408v1 Announce Type: cross Abstract: We extend activation steering to diffusion language models (DLMs) and study a novel problem that arose due to the inference mechanism of DLMs: Modifying a text in-place to manifest a different concept.
By Ryandito Diandaru, Ikhlasul Akmal Hanif, Fadli Aulawi Al Ghiffari, Ahmed Elshabrawy, Alham Fikri Aji
The paper introduces Adjacent Possible Exploration (APE), a selective fine‑tuning strategy for large language models that evaluates multiple parameter updates on small data subsets and accepts only those that surpass a performance threshold. APE draws on evolutionary optimization ideas to filter out destabilizing changes, contrasting with conventional single‑gradient fine‑tuning. Experiments on news summarization show a 33.9% BLEU gain and a 36.2% perplexity drop while keeping computational demands low.
By Javier Mar\'in
arXiv:2605.28456v2 Announce Type: replace
Abstract: Existing Visual Speech Recognition (VSR) systems commonly rely on left-to-right autoregressive decoding, which can force premature decisions on vis...
By Jeong Hun Yeo, Chae Won Kim, Hyeongseop Rha, Yong Man Ro
BiMoGen introduces a unified masked discrete diffusion framework for bidirectional motion‑text generation, addressing limitations of autoregressive models by allowing iterative bidirectional prediction. The method employs a two‑stage training strategy—Decoupled Uni‑ and Cross‑Modal Training followed by supervised fine‑tuning—to establish cross‑modal correspondence and specialize for bidirectional generation. To mitigate train‑test mismatch, Generation‑Aware Self‑Correction exposes the model to its own predictions during training and applies correction passes early in sampling to refine unmasked tokens, achieving competitive results on HumanML3D and KIT‑ML.
By Wanjiang Weng, Yongliang Wu, Xiaofeng Tan, Xingyu Zhu, Wenbo Zhu, Hongsong Wang
BiMoGen introduces a unified masked discrete diffusion framework for bidirectional motion‑text generation, addressing the limitations of autoregressive models in capturing bidirectional dependencies between language and motion. The approach employs a two‑stage training strategy—decoupled uni‑ and cross‑modal pretraining followed by supervised fine‑tuning—to establish robust cross‑modal correspondence, and incorporates Generation‑Aware Self‑Correction to mitigate error propagation during inference. Experiments on HumanML3D and KIT‑ML show competitive performance on both text‑to‑motion and motion‑to‑text tasks, demonstrating the effectiveness of the proposed training and correction mechanisms.
arXiv:2606. 05494v1 Announce Type: cross Abstract: Automatic text summarization has become increasingly important due to the rapid growth of digital textual information.
By Ahmed Alansary, Ali Hamdi
Visual Information-Guided Parallel Decoding for Diffusion Multimodal Large Language Models introduces the VIG‑Sampler, a method that prioritizes tokens for decoding based on their attention to image tokens and penalizes redundancy in image‑attention distributions. The approach aims to improve the quality of multimodal generation by selecting more informative tokens during diffusion decoding. Experiments on seven captioning and VQA benchmarks with three open‑source dMLLMs show that VIG‑Sampler outperforms the Info‑Gain Sampler by an average of 19.3 CIDEr points and achieves better COCO Caption results using only half as many decoding steps.
By Insu Lee, Wooje Park, Wonseok Shin, Jinwoo Son, Byonghyo Shim
The paper introduces Seq-Drifting, a new framework that enables diffusion language models to generate an entire text sequence in a single forward pass. By applying a drift field to continuous token representations, the model learns a direct mapping from Gaussian noise to text, using attraction to guide outputs toward contextually plausible tokens and repulsion to prevent repetition and collapse. Experiments show that Seq-Drifting achieves a perplexity of 36.01 on LM1B and performs competitively in unconditional, conditional, translation, summarization, and reasoning tasks.
By Jiayu Liu, Ruizhi Zhao, Tianyun Ji, Yee Whye Teh, Zhenya Huang, Ning Miao
arXiv:2604. 10024v2 Announce Type: replace-cross Abstract: Long video summarization presents significant challenges for multimodal large language models (MLLMs), particularly in maintaining temporal fidelity over extended durations and producing summaries that are both semantically and temporally grounded.
By Alkesh Patel, Melis Ozyildirim, Ying-Chang Cheng, Ganesh Nagarajan
Diffusion large language models (dLLMs) offer an efficient alternative to autoregressive models through parallel decoding, yet existing post-training methods largely rely on random masking strategies that overlook intrinsic token dependencies. In this work, we present an empirical analysis of attention in dLLMs and show that tokens attending more strongly to unmasked context exhibit greater generation stability and play a critical role in reasoning.