← Back to all news
arXiv Computation and Language By Koshiro Saito, Diasuke Oba, Danushka Bollegala, Naoaki Okazaki

Beyond Model Weights: Sampling-Dependent Fairness in Diffusion Language Models

Read the original on arXiv Computation and Language →

Diffusion language models (DLMs) are gaining traction as an alternative to autoregressive language models, but their fairness has not been well studied. This work investigates how sampling controls—such as block length and remasking strategy—affect social bias in three DLMs (LLaDA, LLaDA‑1.5, and Dream on Open‑BBQ). The authors find that both the magnitude of bias and the relative ranking of models vary significantly with sampling settings, and that even after applying mitigation strategies from autoregressive models, sampling dependence remains, indicating that fairness in DLMs cannot be assessed independently of sampling configuration.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computation and Language.

Related stories

arXiv Machine Learning
Aug 4

The Illusion of Stochasticity in LLMs

arXiv:2604. 06543v2 Announce Type: replace-cross Abstract: In this work, we demonstrate that reliable stochastic sampling is a fundamental yet unfulfilled requirement for Large Language Models (LLMs) operating as agents.

By Xiangming Gu, Soham De, Michalis Titsias, Larisa Markeeva, Petar Veli\v{c}kovi\'c, Razvan Pascanu
llmsagentsreinforcement-learning
More like this →
arXiv Machine Learning
Oct 2

Fork-dLLM: Avoiding the Flexibility Trap in Diffusion Language Models

The paper introduces Fork-dLLM, a hybrid sampling strategy for diffusion language models that combines parallel token generation with autoregressive ordering only at uncertain fallback steps, thereby avoiding the flexibility trap that reduces generation diversity. It also presents ForkGRPO, an extension that applies the GRPO objective selectively during post‑training rollouts, preserving policy‑likelihood ratios while cutting rollout and optimization costs. Experiments show Fork-dLLM achieves pass@k scaling comparable to autoregressive sampling but 2–3× more efficient, and ForkGRPO matches or surpasses AR‑based GRPO baselines at significantly lower training expense.

By Stipe Frkovi\'c, Metod Jazbec, Christian A. Naesseth
llmsdiffusionreinforcement-learning
More like this →
arXiv AI
Oct 28

Fairness Definitions in Language Models Explained

The paper "Fairness Definitions in Language Models Explained" surveys and clarifies various fairness notions applicable to language models (LMs). It introduces a taxonomy categorizing these concepts by transformer architecture—encoder-only, decoder-only, and encoder-decoder—and demonstrates each definition through experiments to illustrate practical implications. The authors also discuss current research challenges and open questions to encourage further innovation in the field.

By Zhipeng Yin, Zichong Wang, Avash Palikhe, Wenbin Zhang
llmsnlpsafety
More like this →
arXiv Machine Learning
Sep 11

Continuous Diffusion Scales Competitively with Discrete Diffusion for Language

The paper revisits the continuous diffusion language model Plaid and introduces RePlaid, aligning its architecture with modern discrete diffusion models. RePlaid achieves a compute gap of only 20× compared to autoregressive models, surpasses Duo with fewer parameters, and outperforms MDLM in over‑trained settings. On OpenWebText, RePlaid sets a new state‑of‑the‑art continuous diffusion perplexity of 22.1 and demonstrates superior generation quality, while theoretical analysis links likelihood‑based training to linear cross‑entropy over time and structured embedding geometries.

By Zhihan Yang, Wei Guo, Shuibai Zhang, Subham Sekhar Sahoo, Yongxin Chen, Arash Vahdat, Morteza Mardani, John Thickstun
llmsragdiffusionbenchmarks
More like this →