Consistency Without Alignment: Item-Sensitive Language Models Indistinguishable From Random

The paper investigates item-sensitivity—whether a language model’s choice depends on the specific input—in a forced-choice signalling task derived from the board game Deception: Murder in Hong Kong. Across seven models, two families, a post‑training ablation, and three scoring rules, every tested cell shows item‑sensitivity, yet many are statistically indistinguishable from random choice and some perform worse than random. The authors term this phenomenon "consistency without alignment" and argue it undermines evaluations that rely solely on item‑sensitivity, permutation consistency, or self‑consistency without an independent reference.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

• llms
• benchmarks
• safety