arXiv Machine Learning August 31, 2026 By Erik Larsen

The Instability of Safety: How Random Seeds and Temperature Expose Inconsistent LLM Refusal Behavior

Read the original on arXiv Machine Learning →

The paper challenges the assumption that large language models (LLMs) produce deterministic safety responses by examining how random seeds and temperature settings affect refusal decisions. Across four instruction‑tuned models and 876 harmful prompts, 18‑28% of prompts flipped between refusal and compliance depending on sampling configuration, with higher temperatures reducing decision stability. The authors introduce a Safety Stability Index (SSI) and recommend multi‑sample evaluation protocols that account for stochastic variation rather than relying on single‑shot tests.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

arXiv Machine Learning

Jun 26

Necessary but Not Sufficient: Temperature Control and Reproducibility in LLM-as-Judge Safety Evaluations

arXiv:2606. 26185v1 Announce Type: new Abstract: LLM-as-judge ("grader") components are now standard in evaluation harnesses, including safety evaluations where a pass/fail verdict may gate downstream deployment decisions.

By Hiroki Tamba

llms safety

arXiv AI

Jun 30

The Joint Effect of Quantization and Sampling Temperature on LLM Safety Alignment: A Factorial Analysis

arXiv:2606. 29581v1 Announce Type: cross Abstract: Modern LLM deployments routinely compress models and raise sampling temperature to reduce cost, latency, or repetition, yet safety evaluations usually treat these choices as fixed implementation details.

By Hari Prasad, Ritam Pal

llms efficiency multimodal safety

arXiv AI

Aug 5

Single Canonical Prompts Underestimate LLM Safety's Surface-Form Sensitivity

arXiv:2608. 02665v1 Announce Type: cross Abstract: A benchmark score is a measurement instrument, yet most benchmarks read each item at a single canonical surface form.

By Yongxi Zhou, Junwei Yao, Yuanzhe Liu, Zihan Dong, Wenbo Ye, Jiaxi Wen, Lai Yun Choi

llms benchmarks safety

Hugging Face Trending Papers

Jun 28

The Joint Effect of Quantization and Sampling Temperature on LLM Safety Alignment: A Factorial Analysis

Modern LLM deployments routinely compress models and raise sampling temperature to reduce cost, latency, or repetition, yet safety evaluations usually treat these choices as fixed implementation details. This leaves a practical uncertainty: does a model that is safe at FP16 and greedy decoding remain safe after it is quantized and sampled stochastically, or do the two deployment knobs amplify one another?

llms efficiency multimodal safety

arXiv Machine Learning

1d ago

Safety-Flag: A Unified Benchmark for the Reliability and Calibration of LLM Content Moderators

arXiv:2609.19072v1 Announce Type: cross Abstract: Large language models are increasingly used for content moderation, but most evaluations still report aggregate accuracy on individual benchmarks. We...

By Yibo Hu

llms benchmarks safety

arXiv Machine Learning

Jun 25

Speculative Decoding at Temperature Zero: A Scoped Safety-Invariance Screen with a 48,072-Sample Expansion

arXiv:2606. 25097v1 Announce Type: new Abstract: Speculative decoding accelerates inference by letting a draft model propose tokens for a target model to verify, raising a concrete safety question: at temperature zero, can draft-side behavior leak into safety-scored outputs?

By Sahil Kadadekar

reinforcement-learning benchmarks safety

One email a morning, machine-written

Related stories

Necessary but Not Sufficient: Temperature Control and Reproducibility in LLM-as-Judge Safety Evaluations

The Joint Effect of Quantization and Sampling Temperature on LLM Safety Alignment: A Factorial Analysis

Single Canonical Prompts Underestimate LLM Safety's Surface-Form Sensitivity

The Joint Effect of Quantization and Sampling Temperature on LLM Safety Alignment: A Factorial Analysis

Safety-Flag: A Unified Benchmark for the Reliability and Calibration of LLM Content Moderators

Speculative Decoding at Temperature Zero: A Scoped Safety-Invariance Screen with a 48,072-Sample Expansion