Typhoon is a gradient‑based masking strategy for masked language models that estimates token importance using the MLM loss gradient and calibrates a masking distribution to match a target budget. The authors evaluate Typhoon against random and whole‑word masking on GLUE tasks MRPC and CoLA across three BERT‑family models and find no statistically significant advantage over baseline methods when multiple random seeds are considered. The study concludes that gradient‑based task‑adaptive masking is competitive with, but does not outperform, uniform random masking at the evaluated scale.
arXiv:2303. 15619v2 Announce Type: replace-cross Abstract: The choice of \emph{which} tokens to mask is a central, under-examined design decision in masked language modeling (MLM).
arXiv:2610. 01578v1 Announce Type: new Abstract: Why can masked prediction learn useful representations that unmasked reconstruction misses?
arXiv:2607. 21353v1 Announce Type: new Abstract: Machine unlearning aims to remove the influence of specific training data while preserving model utility.
arXiv:2609.39704v1 Announce Type: cross Abstract: This paper demonstrate that whether masking-based token pruning helps or hurts worst-group robustness can be predicted before deployment, without lab...
This paper demonstrate that whether masking-based token pruning helps or hurts worst-group robustness can be predicted before deployment, without labels or fine-tuning. A systematic study of semantic...
Informed Masking (IM) is a new technique for aligning Diffusion Large Language Models (dLLMs) with Reinforcement Learning (RL). It identifies a systematic upstream/downstream token structure in dLLM rollouts and shows that masking downstream tokens creates better subproblems for likelihood estimation. When integrated into three state‑of‑the‑art dLLM RL methods on LLaDA‑8B‑Instruct, IM yields up to 2.01%, 8.68%, and 5.77% relative average gains on math and planning benchmarks while improving training stability.
arXiv:2505.19893v2 Announce Type: replace Abstract: Large language model pretraining is compute-intensive, yet many tokens contribute marginally to learning, resulting in inefficiency. We introduce E...
arXiv:2606. 12232v1 Announce Type: new Abstract: Masked diffusion language models (dLLMs) have recently emerged as a competitive alternative to autoregressive language models, with the promise of faster inference via parallel token generation.
arXiv:2605. 26436v2 Announce Type: replace-cross Abstract: Discrete masked diffusion language models such as LLaDA generate text through iterative denoising, where mask tokens are progressively replaced with predicted tokens.
Machine unlearning aims to remove the influence of specific training data while preserving model utility. Many state-of-the-art approaches pursue this goal by restricting the forgetting update to a subset of parameters selected through gradient-based saliency.
arXiv:2606. 10829v1 Announce Type: cross Abstract: Masked diffusion language models can reduce inference steps by revealing multiple tokens per denoising iteration, but this parallelism is fragile: positions that are individually confident may be unsafe to commit together when their predictions are coupled.
arXiv:2604. 18738v3 Announce Type: replace Abstract: Diffusion language models (dLLMs) generate text through iterative denoising, filling multiple masked positions at each step.