Typhoon is a gradient‑based masking strategy for masked language models that estimates token importance using the MLM loss gradient and calibrates a masking distribution to match a target budget. The authors evaluate Typhoon against random and whole‑word masking on GLUE tasks MRPC and CoLA across three BERT‑family models and find no statistically significant advantage over baseline methods when multiple random seeds are considered. The study concludes that gradient‑based task‑adaptive masking is competitive with, but does not outperform, uniform random masking at the evaluated scale.
By Bruce Changlong Xu, Muhammed Shahir Abdurrahman, Hashem Elezabi
arXiv:2606. 12232v1 Announce Type: new Abstract: Masked diffusion language models (dLLMs) have recently emerged as a competitive alternative to autoregressive language models, with the promise of faster inference via parallel token generation.
By Stipe Frkovic, Metod Jazbec, Dan Zhang, Christian A. Naesseth, Ilija Bogunovic, Eric Nalisnick
arXiv:2605. 26436v2 Announce Type: replace-cross Abstract: Discrete masked diffusion language models such as LLaDA generate text through iterative denoising, where mask tokens are progressively replaced with predicted tokens.
By Lin Yao
arXiv:2604. 18738v3 Announce Type: replace Abstract: Diffusion language models (dLLMs) generate text through iterative denoising, filling multiple masked positions at each step.
By Lin Yao
Masked diffusion language models (dLLMs) have recently emerged as a competitive alternative to autoregressive language models, with the promise of faster inference via parallel token generation. A notable limitation of the masked formulation, however, is that once a token has been unmasked it can no longer be revised, leaving dLLMs vulnerable to early sampling mistakes.
arXiv:2505.19893v2 Announce Type: replace
Abstract: Large language model pretraining is compute-intensive, yet many tokens contribute marginally to learning, resulting in inefficiency. We introduce E...
By Melis Ilayda Bal, Volkan Cevher, Michael Muehlebach
The paper introduces TIPS, a tiny inference-time policy that controls which tokens are visible during each denoising pass of masked diffusion language models. TIPS, a 600‑K‑parameter shallow transformer, learns to unmask and remask tokens while keeping the language model frozen, using visibility‑aware confidence features and a fixed schedule for expected visibility. Trained via supervised warmup and reinforcement learning, TIPS achieves over 58% accuracy on GSM8K and MATH‑500 with only 32 decoder evaluations, surpassing strong baselines by more than 11 percentage points and remaining competitive at higher budgets without retraining.
By Ya-Shi Zhang, Jian Tang, Pierre-Andr\'e No\"el, Oleksiy Ostapenko
arXiv:2610. 01578v1 Announce Type: new Abstract: Why can masked prediction learn useful representations that unmasked reconstruction misses?
By Jorge Medina Moreira, Lorenzo Bardone, Lenka Zdeborov\'a
Informed Masking (IM) is a new technique for aligning Diffusion Large Language Models (dLLMs) with Reinforcement Learning (RL). It identifies a systematic upstream/downstream token structure in dLLM rollouts and shows that masking downstream tokens creates better subproblems for likelihood estimation. When integrated into three state‑of‑the‑art dLLM RL methods on LLaDA‑8B‑Instruct, IM yields up to 2.01%, 8.68%, and 5.77% relative average gains on math and planning benchmarks while improving training stability.
By Xiaoyi Yu, Enver Sangineto, Pei Fu, Fiorenzo Parascandolo, Wenhui Tan, Ruikang Zhang, Rita Cucchiara, Ruihua Song, Jian Luan
arXiv:2606. 10829v1 Announce Type: cross Abstract: Masked diffusion language models can reduce inference steps by revealing multiple tokens per denoising iteration, but this parallelism is fragile: positions that are individually confident may be unsafe to commit together when their predictions are coupled.
By Yusuf Sahin, Ahmed Rockey Saikia, Volkan Cevher, Paolo Favaro
arXiv:2609.39704v1 Announce Type: cross
Abstract: This paper demonstrate that whether masking-based token pruning helps or hurts worst-group robustness can be predicted before deployment, without lab...
By Muhammad Zawish, Steven Davy
arXiv:2609.05916v1 Announce Type: cross
Abstract: Large vision-language models (LVLMs) achieve strong multimodal understanding, but the hundreds to thousands of visual tokens they process impose subs...
By Yichen Guo, Tinghao Wang, Qizhe Zhang, Lingbei Meng, Yuan Zhang, Jiajun Cao, Hao Jiang, Chenwei Wu, Jixian Wu, Sixiang Chen, Tao Luo, Hongyang Cheng, Kai Tang, Chenxi Li, Renyuan Li, Xiande Huang, Wenya Wang, Shanghang Zhang