arXiv:2303. 15619v2 Announce Type: replace-cross Abstract: The choice of \emph{which} tokens to mask is a central, under-examined design decision in masked language modeling (MLM).
Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.
Typhoon is a gradient‑based masking strategy for masked language models that estimates token importance using the MLM loss gradient and calibrates a masking distribution to match a target budget. The authors evaluate Typhoon against random and whole‑word masking on GLUE tasks MRPC and CoLA across three BERT‑family models and find no statistically significant advantage over baseline methods when multiple random seeds are considered. The study concludes that gradient‑based task‑adaptive masking is competitive with, but does not outperform, uniform random masking at the evaluated scale.
arXiv:2606. 12232v1 Announce Type: new Abstract: Masked diffusion language models (dLLMs) have recently emerged as a competitive alternative to autoregressive language models, with the promise of faster inference via parallel token generation.
arXiv:2605. 26436v2 Announce Type: replace-cross Abstract: Discrete masked diffusion language models such as LLaDA generate text through iterative denoising, where mask tokens are progressively replaced with predicted tokens.
arXiv:2604. 18738v3 Announce Type: replace Abstract: Diffusion language models (dLLMs) generate text through iterative denoising, filling multiple masked positions at each step.
Masked diffusion language models (dLLMs) have recently emerged as a competitive alternative to autoregressive language models, with the promise of faster inference via parallel token generation. A notable limitation of the masked formulation, however, is that once a token has been unmasked it can no longer be revised, leaving dLLMs vulnerable to early sampling mistakes.
arXiv:2505.19893v2 Announce Type: replace Abstract: Large language model pretraining is compute-intensive, yet many tokens contribute marginally to learning, resulting in inefficiency. We introduce E...