← Back to all news
arXiv AI By Bruce Changlong Xu, Muhammed Shahir Abdurrahman, Hashem Elezabi

Typhoon: Gradient-Based Task-Specific Masking for Masked Language Models

Read the original on arXiv AI →

Typhoon is a gradient‑based masking strategy for masked language models that estimates token importance using the MLM loss gradient and calibrates a masking distribution to match a target budget. The authors evaluate Typhoon against random and whole‑word masking on GLUE tasks MRPC and CoLA across three BERT‑family models and find no statistically significant advantage over baseline methods when multiple random seeds are considered. The study concludes that gradient‑based task‑adaptive masking is competitive with, but does not outperform, uniform random masking at the evaluated scale.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

Related stories

arXiv Computation and Language
Sep 23

Informed Masking: Structure-Aware Perturbation for Reinforcement Learning in Diffusion Large Language Models

Informed Masking (IM) is a new technique for aligning Diffusion Large Language Models (dLLMs) with Reinforcement Learning (RL). It identifies a systematic upstream/downstream token structure in dLLM rollouts and shows that masking downstream tokens creates better subproblems for likelihood estimation. When integrated into three state‑of‑the‑art dLLM RL methods on LLaDA‑8B‑Instruct, IM yields up to 2.01%, 8.68%, and 5.77% relative average gains on math and planning benchmarks while improving training stability.

By Xiaoyi Yu, Enver Sangineto, Pei Fu, Fiorenzo Parascandolo, Wenhui Tan, Ruikang Zhang, Rita Cucchiara, Ruihua Song, Jian Luan
llmsdiffusionreinforcement-learningbenchmarks
More like this →