The paper investigates how model size and quantization affect social bias and abstention in question answering. Using the BBQ benchmark and models from Qwen3, Gemma-3, and SmolLM2, it finds that larger models improve accuracy but bias does not scale simply, while quantization effects differ across settings. Importantly, many configurations with fewer biased responses also show increased abstention, suggesting that lower bias may partly result from more frequent abstention.
Almost every large language model that reaches a broad audience is quantized: trained in full precision, then compressed for efficiency. This step is assumed harmless and its safety is rarely re-checked.
Post‑training quantization of large language models reduces memory usage but can alter social biases in ways that aggregate metrics miss. In a large‑scale study of 50 quantized models on PostTrainingBiasBench, the authors discovered a phenomenon called quantization‑induced bias flipping, where up to 21% of responses switch from biased to unbiased or vice versa, especially for uncertain predictions and stronger quantization (4‑bit vs 8‑bit). These flips lead to asymmetric impacts across demographic groups, with some groups experiencing up to an 18.6% worsening of bias while others improve by 14.1%, resulting in misleadingly neutral overall scores.
The paper compares two common ways of evaluating large language models (LLMs): prompting them to answer questions directly and scoring candidate answers using likelihood-based metrics. The authors introduce a new protocol that ranks declarative statements derived from question–answer pairs, and test it across 95 decoder-only models (0.1B–104B parameters) on 10 multiple-choice QA datasets. They find that while prompted answering accuracy improves sharply with model scale and instruction tuning, statement‑likelihood ranking accuracy stays relatively stable, indicating that the two evaluation methods probe different aspects of model behavior.
arXiv:2607. 08734v1 Announce Type: new Abstract: Post-training quantization is widely used to deploy large language models in resource-constrained settings, yet its evaluation relies almost exclusively on accuracy and perplexity.
Post-training quantization is widely used to deploy large language models in resource-constrained settings, yet its evaluation relies almost exclusively on accuracy and perplexity. We show that these metrics fail to capture behavioral changes induced by quantization.
arXiv:2610.08559v1 Announce Type: cross Abstract: Activation steering has gained popularity as a lightweight inference-time debiasing technique for large language models. However, prior work reports...
The paper investigates how quantization affects large language models’ self‑explanations, examining natural language explanations and counterfactual examples across three quantization techniques and bit widths. Results show moderate declines in explanation quality (up to 4.4%) and faithfulness (up to 3.9%), with user studies indicating up to an 8.5% drop in coherence and trustworthiness. Larger models are less resilient in quality but remain more faithful, and no single quantization method consistently outperforms others across accuracy, quality, and faithfulness.
arXiv:2604. 03904v2 Announce Type: replace-cross Abstract: Large language models (LLMs) often produce confident but incorrect answers, in part because standard evaluation incentives reward guessing over expressing uncertainty.
The paper introduces a new evaluation setting called penalty‑framed no‑valid‑option MCQA, where multiple‑choice questions may contain no correct answer. By removing the correct option from the MMLU‑Pro mathematics subset and allowing models to either pick an option or abstain, the authors penalize forced‑choice responses that are invalid. Experiments reveal that even models with high standard MCQA accuracy can still produce invalid forced‑choice answers, indicating that traditional accuracy metrics miss an important aspect of model reliability.
The paper investigates test-time scaling for small-scale open-ended reasoning models by introducing semantic cluster voting, a judge-free method that groups model outputs by meaning and selects the largest cluster’s representative. Experiments on a 250-prompt benchmark across three 3‑B models show that voting improves accuracy on math tasks (10–29 point gains) but only marginally on open-ended text (+0.30 points). A larger 200‑problem MATH‑500 run confirms a +8.5 point math improvement, while most open-ended samples fall into a single cluster, indicating limited diversity and that higher decoding temperatures reduce quality.
arXiv:2610.18343v1 Announce Type: cross Abstract: Post-training quantisation reduces the memory cost of large language models, but it remains unclear whether compression preserves language skills of...
arXiv:2602. 23971v4 Announce Type: replace-cross Abstract: Sycophancy, the tendency of large language models to favour user-affirming responses over critical engagement, has been identified as an alignment failure, particularly in high-stakes advisory and social contexts.