arXiv AI September 25, 2026 By Yu Sun, Junhao Xu, Jiajia Shi, Zijin Yang

Type-Safe Is Not Error-Free: A Constrained Decision Head Follows the Option Name, Not the Rubric Bound to It

The study examines how renaming option labels in typed decision models affects model behavior. By swapping the names of two options (e.g., from 0/1 to no/yes) while keeping the underlying rubrics unchanged, the authors observed a dramatic shift in decision rankings—AUC dropped from .94 to .23 and answer flips increased by 70.4 per hundred. The effect is amplified with more options and depends on the semantic polarity of the labels, yet the models still maintain a zero type‑error rate.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

Hugging Face Trending Papers

Sep 22

Type-Safe Is Not Error-Free: A Constrained Decision Head Follows the Option Name, Not the Rubric Bound to It

Typed decision models are built for settings where model outputs are consumed directly by software. Instead of generating free-form text, they return a decision over a predefined set of options. By co...

arXiv AI

Sep 15

Rubrics as an Attack Surface: Stealthy Preference Drift in LLM Judges

arXiv:2602.13576v2 Announce Type: replace-cross Abstract: Evaluation and alignment pipelines for large language models increasingly rely on LLM-based judges, whose behavior is guided by natural-langu...

By Ruomeng Ding, Yifei Pang, He Sun, Yizhong Wang, Zhiwei Steven Wu, Zhun Deng

llms benchmarks safety

arXiv AI

Aug 5

Single Canonical Prompts Underestimate LLM Safety's Surface-Form Sensitivity

arXiv:2608. 02665v1 Announce Type: cross Abstract: A benchmark score is a measurement instrument, yet most benchmarks read each item at a single canonical surface form.

By Yongxi Zhou, Junwei Yao, Yuanzhe Liu, Zihan Dong, Wenbo Ye, Jiaxi Wen, Lai Yun Choi

llms benchmarks safety

arXiv AI

2d ago

Evaluating and Benchmarking the System One Model Jev

arXiv:2609.37647v1 Announce Type: cross Abstract: Jev is a commercial System One model from TypeSafe AI that does not generate text: given a state and typed questions, it returns a choice from fixed...

By Tobias Deu{\ss}er, Lorenz Sparrenberg, Rafet Sifa

benchmarks

arXiv Computation and Language

Sep 25

JevOut: Natural Context Can Flip Decision Models

JevOut demonstrates that natural, short additions to the context of decision models can flip their outputs from correct to incorrect, even when the correct answer remains unchanged. By optimizing context additions while keeping the source, question, choices, and gold answer fixed, the study found that 61.4% of initially correct decisions were redirected to a wrong option, with 45% receiving high confidence. Similar fragility was observed across three other decision systems on seven datasets, with flip rates between 64.9% and 73.2%.

By Zixiang Xu

arXiv AI

Jul 28

Tag Questions and the Generational Reversal of Sycophancy Across 45 Language Models

arXiv:2607. 23976v1 Announce Type: cross Abstract: Appending a two-word confirmation tag to a decision question -- "Is X the better choice?

By Tapan Parikh

llms rag benchmarks

One email a morning, machine-written

Related stories

Type-Safe Is Not Error-Free: A Constrained Decision Head Follows the Option Name, Not the Rubric Bound to It

Rubrics as an Attack Surface: Stealthy Preference Drift in LLM Judges

Single Canonical Prompts Underestimate LLM Safety's Surface-Form Sensitivity

Evaluating and Benchmarking the System One Model Jev

JevOut: Natural Context Can Flip Decision Models

Tag Questions and the Generational Reversal of Sycophancy Across 45 Language Models