The paper demonstrates that typed decision models can produce type‑correct outputs that nevertheless do not adhere to the explicit definitions of their options. By swapping only the names of options while keeping the definitions, question, and state unchanged, the authors show that decision flips can increase dramatically—up to 70.4 percentage points—when using yes/no names versus neutral controls. This effect is consistent across multiple models and decision rules, and even random strings as names yield flip rates similar to neutral controls, indicating that option‑name polarity contributes to decision instability beyond mere reassignment.
By Yu Sun, Junhao Xu, Jiajia Shi, Zijin Yang
Typed decision models are built for settings where model outputs are consumed directly by software. Instead of generating free-form text, they return a decision over a predefined set of options. By co...
The study examines how renaming option labels in typed decision models affects model behavior. By swapping the names of two options (e.g., from 0/1 to no/yes) while keeping the underlying rubrics unchanged, the authors observed a dramatic shift in decision rankings—AUC dropped from .94 to .23 and answer flips increased by 70.4 per hundred. The effect is amplified with more options and depends on the semantic polarity of the labels, yet the models still maintain a zero type‑error rate.
By Yu Sun, Junhao Xu, Jiajia Shi, Zijin Yang
The paper investigates typed decision models used as guardrails in agent systems, evaluating seven open‑weight models on tasks such as prompt‑injection, jailbreak, and toxic‑content screening. Accuracy for allow‑or‑block decisions ranges from 36% to 72% above chance, with models either over‑allowing or over‑blocking. The authors demonstrate that simple textual manipulations—like adding misleading names or server log lines—can dramatically increase fail‑open rates, and that all tested defenses can be bypassed, suggesting these models are unsuitable as the sole decision component.
By Seyedarmin Azizi, Erfan Baghaei Potraghloo, Massoud Pedram
TypedBench is a new benchmark designed to evaluate system‑one decision models that output calibrated probabilities for typed answers—categorical, ordinal, or binary—using a non‑generative interface. The benchmark assesses models on policy adherence, wording robustness, probability quality (via selective prediction, ordinal proper scoring rules, and calibration error), and the cost of decisions under asymmetric cost matrices, using seven policy‑labelled generators and nine evaluation suites. Experiments on a hosted model, an open encoder, and a range of open decoders (0.8B–9B parameters) reveal that the hosted model is wording‑sensitive and underconfident, while decoders are slow and less accurate on policy questions, highlighting the need for joint evaluation across multiple dimensions.
By Rahul Sharma, Andrew B. Ducan, Ga\'etan Marceau Caron, Sebastian J. Vollmer
The paper evaluates System One decision models—typed models that output probabilities for branching decisions—against supervised classifiers and generative language models on automated decision gate tasks. Eight checkpoints from six families, including the hosted model Jev, were benchmarked on workflow, intent, and social‑science items, showing that small trained classifiers match or slightly outperform decision models on intent and workflow when labels are available, while decision models outperform zero‑shot classifiers when labels are absent. The study also explores calibration, risk thresholds, and cost‑efficiency trade‑offs, providing condition‑dependent design guidelines for automated decision gates.
By Amir Rafe, Subasish Das
arXiv:2610.02267v1 Announce Type: new
Abstract: Agent harnesses make many small, typed decisions per task: which model to call, which tool to use, whether retrieved text is relevant, whether an input...
By Jiawei Li
arXiv:2609.37647v1 Announce Type: cross
Abstract: Jev is a commercial System One model from TypeSafe AI that does not generate text: given a state and typed questions, it returns a choice from fixed...
By Tobias Deu{\ss}er, Lorenz Sparrenberg, Rafet Sifa
Prefill‑only decision models evaluate every candidate in a menu in a single forward pass, avoiding decoding and reducing cost by one to two orders of magnitude compared to generative language models. The paper demonstrates that when only the candidate menu changes, the model’s post‑intervention accuracy can be predicted solely from the cached first‑pass distribution using a simple estimator that renormalizes and selects the argmax, without any labels or second pass. Across seven model families, ten datasets, and two task types, this menu‑only intervention prediction is within 4.2 points of actual accuracy, and in one family it is exact, whereas a probability‑level variant fails by 21 points, indicating the property resides in ranking rather than calibrated probabilities.
By Ran Li, Lei Chen
arXiv:2609.23886v1 Announce Type: new
Abstract: Software delegates more of its branches to models every year: which queue a ticket enters, whether a command is safe to run, whether a claim clears wit...
By Zehua Cheng, Wei Dai, Jiahao Sun
arXiv:2609.39496v1 Announce Type: cross
Abstract: Typed decision models such as Jev offer an efficient alternative to generative LLMs in decision-making workflows by selecting directly from predefine...
By Jike Zhong, Ming Li, Yuxiang Lai
The report introduces AnyJev, a method that extracts typed decisions from a pretrained instruction‑tuned language model by restricting the next‑token distribution to a set of option tokens. It corrects two defects—label bias and position bias—without any gradient updates by dividing out a label prior and averaging log‑probabilities over cyclic rotations of the option list. Experiments on two 20‑option tasks show that rotations reduce order‑flip rates and improve accuracy across all tested models, while a stopping rule reduces the number of required rotations and increases decision throughput.
By Jiamu Zhang, Tianze Yang, Yucheng Shi, Evan Chen, Zixiang Nie, Kelly Wan, Liangjie Hong, Ninghao Liu, Liang Wu