The study examines how renaming option labels in typed decision models affects model behavior. By swapping the names of two options (e.g., from 0/1 to no/yes) while keeping the underlying rubrics unchanged, the authors observed a dramatic shift in decision rankings—AUC dropped from .94 to .23 and answer flips increased by 70.4 per hundred. The effect is amplified with more options and depends on the semantic polarity of the labels, yet the models still maintain a zero type‑error rate.
By Yu Sun, Junhao Xu, Jiajia Shi, Zijin Yang
Typed decision models are built for settings where model outputs are consumed directly by software. Instead of generating free-form text, they return a decision over a predefined set of options. By co...
arXiv:2610.02586v1 Announce Type: new
Abstract: A typed decision model answers a fixed question about an input by returning a probability for each of several caller-defined options. Each option carri...
By Seyedarmin Azizi, Erfan Baghaei Potraghloo, Massoud Pedram
TypedBench is a new benchmark designed to evaluate system‑one decision models that output calibrated probabilities for typed answers—categorical, ordinal, or binary—using a non‑generative interface. The benchmark assesses models on policy adherence, wording robustness, probability quality (via selective prediction, ordinal proper scoring rules, and calibration error), and the cost of decisions under asymmetric cost matrices, using seven policy‑labelled generators and nine evaluation suites. Experiments on a hosted model, an open encoder, and a range of open decoders (0.8B–9B parameters) reveal that the hosted model is wording‑sensitive and underconfident, while decoders are slow and less accurate on policy questions, highlighting the need for joint evaluation across multiple dimensions.
By Rahul Sharma, Andrew B. Ducan, Ga\'etan Marceau Caron, Sebastian J. Vollmer
JevOut demonstrates that natural, short additions to the context of decision models can flip their outputs from correct to incorrect, even when the correct answer remains unchanged. By optimizing context additions while keeping the source, question, choices, and gold answer fixed, the study found that 61.4% of initially correct decisions were redirected to a wrong option, with 45% receiving high confidence. Similar fragility was observed across three other decision systems on seven datasets, with flip rates between 64.9% and 73.2%.
By Zixiang Xu
The paper evaluates System One decision models—typed models that output probabilities for branching decisions—against supervised classifiers and generative language models on automated decision gate tasks. Eight checkpoints from six families, including the hosted model Jev, were benchmarked on workflow, intent, and social‑science items, showing that small trained classifiers match or slightly outperform decision models on intent and workflow when labels are available, while decision models outperform zero‑shot classifiers when labels are absent. The study also explores calibration, risk thresholds, and cost‑efficiency trade‑offs, providing condition‑dependent design guidelines for automated decision gates.
By Amir Rafe, Subasish Das
The paper introduces a paired candidate‑coverage benchmark protocol for typed decision models, evaluating two models—Laya and Jev—on datasets such as AG News, DBpedia, Emotion, and TREC. It reports that Laya detects a high percentage of missing-answer cases but also falsely rejects many valid candidates, whereas Jev shows lower false rejection rates but also lower detection of missing answers. The study highlights the need for separate measurements of classification, score ranking, and rejection policies, noting that the benchmark is descriptive and limited to reference‑label omission.
By Jiawen Lu, Tongtong Wu
arXiv:2609.39496v1 Announce Type: cross
Abstract: Typed decision models such as Jev offer an efficient alternative to generative LLMs in decision-making workflows by selecting directly from predefine...
By Jike Zhong, Ming Li, Yuxiang Lai
The paper investigates typed decision models used as guardrails in agent systems, evaluating seven open‑weight models on tasks such as prompt‑injection, jailbreak, and toxic‑content screening. Accuracy for allow‑or‑block decisions ranges from 36% to 72% above chance, with models either over‑allowing or over‑blocking. The authors demonstrate that simple textual manipulations—like adding misleading names or server log lines—can dramatically increase fail‑open rates, and that all tested defenses can be bypassed, suggesting these models are unsuitable as the sole decision component.
By Seyedarmin Azizi, Erfan Baghaei Potraghloo, Massoud Pedram
The paper demonstrates that multiple‑choice visual question answering (MC‑VQA) benchmarks are unreliable because model performance is highly sensitive to semantically neutral prompt formatting choices—such as option ID sets, delimiters, and separators—despite protocols that mitigate option‑order effects. Across seven multimodal large language models and five datasets, the authors observed frequent rank reversals when systematically varying 48 equivalent prompt formats, attributing the instability to tokenizer‑induced token fusion or removal and to how option ID sets influence attention patterns. Consequently, MC‑VQA rankings correlate weakly with open‑ended evaluation, revealing that MC‑VQA reflects option‑selection dynamics as well as multimodal reasoning.
By Fabio Rosenthal, Sebastian Schmidt, Thorsten Graf, Thorsten Bagdonat, Stephan G\"unnemann, Leo Schwinn
arXiv:2608. 02665v1 Announce Type: cross Abstract: A benchmark score is a measurement instrument, yet most benchmarks read each item at a single canonical surface form.
By Yongxi Zhou, Junwei Yao, Yuanzhe Liu, Zihan Dong, Wenbo Ye, Jiaxi Wen, Lai Yun Choi
arXiv:2610.02267v1 Announce Type: new
Abstract: Agent harnesses make many small, typed decisions per task: which model to call, which tool to use, whether retrieved text is relevant, whether an input...
By Jiawei Li