Type-Safe Is Not Error-Free: A Constrained Decision Head Follows the Option Name, Not the Rubric Bound to It
Read the original on Hugging Face Trending Papers →The Flow has not summarised this story yet — read it at Hugging Face Trending Papers.
The Flow has not summarised this story yet — read it at Hugging Face Trending Papers.
The study examines how renaming option labels in typed decision models affects model behavior. By swapping the names of two options (e.g., from 0/1 to no/yes) while keeping the underlying rubrics unchanged, the authors observed a dramatic shift in decision rankings—AUC dropped from .94 to .23 and answer flips increased by 70.4 per hundred. The effect is amplified with more options and depends on the semantic polarity of the labels, yet the models still maintain a zero type‑error rate.
The paper demonstrates that typed decision models can produce type‑correct outputs that nevertheless do not adhere to the explicit definitions of their options. By swapping only the names of options while keeping the definitions, question, and state unchanged, the authors show that decision flips can increase dramatically—up to 70.4 percentage points—when using yes/no names versus neutral controls. This effect is consistent across multiple models and decision rules, and even random strings as names yield flip rates similar to neutral controls, indicating that option‑name polarity contributes to decision instability beyond mere reassignment.
arXiv:2610.02586v1 Announce Type: new Abstract: A typed decision model answers a fixed question about an input by returning a probability for each of several caller-defined options. Each option carri...
arXiv:2610.16655v1 Announce Type: new Abstract: Decision APIs that return a probability for each option are routinely used as if those probabilities were properties of the options themselves. That re...
arXiv:2602.13576v2 Announce Type: replace-cross Abstract: Evaluation and alignment pipelines for large language models increasingly rely on LLM-based judges, whose behavior is guided by natural-langu...
arXiv:2608. 02665v1 Announce Type: cross Abstract: A benchmark score is a measurement instrument, yet most benchmarks read each item at a single canonical surface form.