Large language models (LLMs) compression reduces deployment costs, but standard aggregate metrics like perplexity and accuracy often mask underlying behavioral shifts. In this work, we systematically evaluate 3 LLMs across 11 compression methods to investigate the effects of compression on knowledge retention, model confidence, and social bias.
arXiv:2608. 19670v1 Announce Type: new Abstract: Large language models (LLMs) compression reduces deployment costs, but standard aggregate metrics like perplexity and accuracy often mask underlying behavioral shifts.
Post‑training quantization of large language models reduces memory usage but can alter social biases in ways that aggregate metrics miss. In a large‑scale study of 50 quantized models on PostTrainingBiasBench, the authors discovered a phenomenon called quantization‑induced bias flipping, where up to 21% of responses switch from biased to unbiased or vice versa, especially for uncertain predictions and stronger quantization (4‑bit vs 8‑bit). These flips lead to asymmetric impacts across demographic groups, with some groups experiencing up to an 18.6% worsening of bias while others improve by 14.1%, resulting in misleadingly neutral overall scores.
The paper introduces a bias depth score to differentiate between stable model preferences (Deep biases) and prompt‑dependent responses (Shallow biases) in large language models. By analyzing 4,442 opinion prompts across four models, it finds that only about a quarter of concentrated preferences persist after scenario reframing, indicating that most are shallow. The study shows Deep biases are more often inherited from pretraining and harder to remove through fine‑tuning or prompt‑based debiasing, highlighting the need to distinguish learned biases from prompt artifacts.
arXiv:2610.08559v1 Announce Type: cross Abstract: Activation steering has gained popularity as a lightweight inference-time debiasing technique for large language models. However, prior work reports...
The paper introduces AttriBench, a benchmark dataset that balances author fame and demographics to study quote attribution in large language models (LLMs). Using AttriBench, the authors evaluate 11 popular LLMs and find that accurate attribution remains difficult, with significant disparities across race, gender, and intersectional groups. They also identify a new failure mode—suppression—where models omit attribution entirely, which is unevenly distributed across demographics and not reflected by standard accuracy metrics.
arXiv:2607. 18269v1 Announce Type: new Abstract: The wisdom of crowds -- the finding that aggregating judgments across individuals often outperforms the best individual -- has been extensively studied with human forecasters.
As Large Language Models are increasingly deployed in critical applications, robustly evaluating their social biases is paramount. However, the current literature suffers from widespread methodological fragmentation, which yields contradictory conclusions.
The paper examines how large language models (LLMs) respond to different demographic cues—such as names—when users seek advice, focusing on race and gender in a U.S. context. It finds that using different cues for the same group leads to only partially overlapping changes in model responses, producing inconsistent conclusions about personalization and unstable bias metrics. The authors argue that LLMs react to linguistic signals tied to cues rather than to stable demographic categories, and they call for evaluations that use multiple cues and consider underlying mechanisms.
The paper introduces Collective Bias Mitigation (CBM), a framework that reduces bias in large language models by learning fine‑grained model behavior and enabling knowledge sharing among diverse LLMs. CBM systematically explores how to select and organize distinct models to produce fairer responses, outperforming standalone baselines. Experiments demonstrate significant bias reduction, with the Committee topology balancing effectiveness and inference cost.
The paper introduces GPTBIAS, a framework that uses powerful large language models like GPT‑4 to evaluate bias in other LLMs. It employs specially crafted prompts called Bias Attack Instructions to probe for bias and outputs a bias score along with detailed information such as bias types, affected demographics, keywords, reasons, and improvement suggestions. Extensive experiments demonstrate the framework’s effectiveness and usability.
arXiv:2606. 24083v1 Announce Type: cross Abstract: "Talk short.
Large language models (LLMs) are increasingly used to assess social bias in text, but the passages they evaluate often contain surface noise such as typos and broken punctuation. This study applied five realistic noise conditions at varying intensities to 3,822 stereotype‑related responses and compared bias judgments on noisy versus original text. The findings show that noise disproportionately turns neutral judgments into biased ones—up to 120 times more likely—while rarely converting biased judgments into neutral ones, and that the most fragile LLM judge exhibits the greatest distortion at mild noise levels. As LLMs become more robust, the bias distortion tends toward parity rather than reversal, meaning bias measured on noisy text is systematically overestimated, especially in fairness‑critical categories.