Searching for "Harmful Refusal": A Psychometric Audit of an AI Safety Benchmark
Read the original on arXiv AI →The paper investigates whether the HELM Safety benchmark’s HarmBench dataset measures a single attribute—harmful refusal—using psychometric methods. Multidimensional item response theory indicates HarmBench does not capture a single attribute, and differential item functioning reveals that models from different developers score items differently, suggesting domain‑specific effects. The authors conclude that safety scores should only be used for attribute comparison after confirming single‑attribute validity, warning that aggregated scores can mask saturation and conflated behaviors.
Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.