Diffusion language models (DLMs) are gaining traction as an alternative to autoregressive language models, but their fairness has not been well studied. This work investigates how sampling controls—such as block length and remasking strategy—affect social bias in three DLMs (LLaDA, LLaDA‑1.5, and Dream on Open‑BBQ). The authors find that both the magnitude of bias and the relative ranking of models vary significantly with sampling settings, and that even after applying mitigation strategies from autoregressive models, sampling dependence remains, indicating that fairness in DLMs cannot be assessed independently of sampling configuration.
arXiv:2609.39560v1 Announce Type: new Abstract: Sampling several responses and voting over their answers can improve a language model's accuracy, but repeated answers limit the benefit of additional...
arXiv:2610.08559v1 Announce Type: cross Abstract: Activation steering has gained popularity as a lightweight inference-time debiasing technique for large language models. However, prior work reports...
arXiv:2604. 06543v2 Announce Type: replace-cross Abstract: In this work, we demonstrate that reliable stochastic sampling is a fundamental yet unfulfilled requirement for Large Language Models (LLMs) operating as agents.
The paper introduces Fork-dLLM, a hybrid sampling strategy for diffusion language models that combines parallel token generation with autoregressive ordering only at uncertain fallback steps, thereby avoiding the flexibility trap that reduces generation diversity. It also presents ForkGRPO, an extension that applies the GRPO objective selectively during post‑training rollouts, preserving policy‑likelihood ratios while cutting rollout and optimization costs. Experiments show Fork-dLLM achieves pass@k scaling comparable to autoregressive sampling but 2–3× more efficient, and ForkGRPO matches or surpasses AR‑based GRPO baselines at significantly lower training expense.
The paper "Fairness Definitions in Language Models Explained" surveys and clarifies various fairness notions applicable to language models (LMs). It introduces a taxonomy categorizing these concepts by transformer architecture—encoder-only, decoder-only, and encoder-decoder—and demonstrates each definition through experiments to illustrate practical implications. The authors also discuss current research challenges and open questions to encourage further innovation in the field.
The paper revisits the continuous diffusion language model Plaid and introduces RePlaid, aligning its architecture with modern discrete diffusion models. RePlaid achieves a compute gap of only 20× compared to autoregressive models, surpasses Duo with fewer parameters, and outperforms MDLM in over‑trained settings. On OpenWebText, RePlaid sets a new state‑of‑the‑art continuous diffusion perplexity of 22.1 and demonstrates superior generation quality, while theoretical analysis links likelihood‑based training to linear cross‑entropy over time and structured embedding geometries.
The paper investigates how model size and quantization affect social bias and abstention in question answering. Using the BBQ benchmark and models from Qwen3, Gemma-3, and SmolLM2, it finds that larger models improve accuracy but bias does not scale simply, while quantization effects differ across settings. Importantly, many configurations with fewer biased responses also show increased abstention, suggesting that lower bias may partly result from more frequent abstention.
arXiv:2607. 25292v1 Announce Type: new Abstract: Silicon sampling uses language models as proxies for human survey respondents, treating each model call as an independent draw from the persona's response distribution.
arXiv:2506. 01928v5 Announce Type: replace-cross Abstract: Diffusion-based language models offer a compelling alternative to autoregressive (AR) models by enabling parallel and controllable generation.
The paper introduces Collective Bias Mitigation (CBM), a framework that reduces bias in large language models by learning fine‑grained model behavior and enabling knowledge sharing among diverse LLMs. CBM systematically explores how to select and organize distinct models to produce fairer responses, outperforming standalone baselines. Experiments demonstrate significant bias reduction, with the Committee topology balancing effectiveness and inference cost.
arXiv:2601.03087v2 Announce Type: replace Abstract: Large Language Models (LLMs) exhibit systematic biases across demographic groups. Auditing is proposed as an accountability tool for black-box LLM...
arXiv:2603.01367v3 Announce Type: replace Abstract: Masked diffusion models (MDMs) are trained to predict tokens in every order but generate in one. As such, MDM likelihood is an intractable sum over...