Hugging Face Trending Papers

Type-Safe Is Not Error-Free: A Constrained Decision Head Follows the Option Name, Not the Rubric Bound to It

arXiv AI
Sep 25

Type-Safe Is Not Error-Free: A Constrained Decision Head Follows the Option Name, Not the Rubric Bound to It

The study examines how renaming option labels in typed decision models affects model behavior. By swapping the names of two options (e.g., from 0/1 to no/yes) while keeping the underlying rubrics unchanged, the authors observed a dramatic shift in decision rankings—AUC dropped from .94 to .23 and answer flips increased by 70.4 per hundred. The effect is amplified with more options and depends on the semantic polarity of the labels, yet the models still maintain a zero type‑error rate.

By Yu Sun, Junhao Xu, Jiajia Shi, Zijin Yang
arXiv AI
Oct 9

Type-Safe Is Not Error-Free: Typed Decision Models Follow the Option Name, Not the Definition Bound to It

The paper demonstrates that typed decision models can produce type‑correct outputs that nevertheless do not adhere to the explicit definitions of their options. By swapping only the names of options while keeping the definitions, question, and state unchanged, the authors show that decision flips can increase dramatically—up to 70.4 percentage points—when using yes/no names versus neutral controls. This effect is consistent across multiple models and decision rules, and even random strings as names yield flip rates similar to neutral controls, indicating that option‑name polarity contributes to decision instability beyond mere reassignment.

By Yu Sun, Junhao Xu, Jiajia Shi, Zijin Yang
arXiv Computation and Language
Nov 4

Swap, Replicate, Read the Mass: A Protocol for Auditing Demographic Bias in System One Decision Models

The paper introduces a comprehensive audit protocol for System One decision models, which provide typed, probability‑scored answers without human oversight. Using three adapted benchmarks—BBQ, CrowS‑Pairs, and SilverSpoon—it evaluates twelve models plus a no‑thinking baseline across 1,638,000 decisions, revealing widespread stereotype alignment, order‑dependent responses, and confidence instability. The study demonstrates that swapping answer options and reversing question order significantly alters model outputs, underscoring the need for order‑invariant scoring in procurement.

By Alejandro Mozo Quesada (Universidad de M\'alaga, Spain), Sergio G\'alvez Rojas (Universidad de M\'alaga, Spain), Francisco L\'opez Valverde (Universidad de M\'alaga, Spain)
arXiv Machine Learning
Oct 3

Benchmarking System One decision models against trained classifiers and language models for automated decision gates

The paper evaluates System One decision models—typed models that output probabilities for branching decisions—against supervised classifiers and generative language models on automated decision gate tasks. Eight checkpoints from six families, including the hosted model Jev, were benchmarked on workflow, intent, and social‑science items, showing that small trained classifiers match or slightly outperform decision models on intent and workflow when labels are available, while decision models outperform zero‑shot classifiers when labels are absent. The study also explores calibration, risk thresholds, and cost‑efficiency trade‑offs, providing condition‑dependent design guidelines for automated decision gates.

By Amir Rafe, Subasish Das
arXiv Computation and Language
Sep 25

JevOut: Natural Context Can Flip Decision Models

JevOut demonstrates that natural, short additions to the context of decision models can flip their outputs from correct to incorrect, even when the correct answer remains unchanged. By optimizing context additions while keeping the source, question, choices, and gold answer fixed, the study found that 61.4% of initially correct decisions were redirected to a wrong option, with 45% receiving high confidence. Similar fragility was observed across three other decision systems on seven datasets, with flip rates between 64.9% and 73.2%.

By Zixiang Xu
arXiv AI
3d ago

Small Heads, Strong Decisions: A Controlled Study of Typed-Decision Models

The paper reports a controlled study of typed‑decision models, re‑implementing five different readout heads on a single backbone (Qwen3‑0.6B) with a unified training protocol. Results show that head complexity has little impact on accuracy, while the backbone size is the dominant factor for performance gains. The study also finds that batching all questions in one prompt yields comparable accuracy with faster inference and training.

By Ritvik Rastogi
arXiv Machine Learning
Oct 8

Input-Blind Controls Produce Substantial Oracle Headroom for Layer Programs in Multiple-Choice Evaluation

The study investigates whether adaptive layer-skipping and repetition programs truly improve language‑model inference or merely exploit input‑blind controls. Using 32 such programs on Qwen3‑4B‑Base and Llama‑3.1‑8B across 4,413 multiple‑choice items, the authors compare real program gains to those from input‑blind perturbations and fixed letter offsets. Results show that input‑blind controls can produce comparable or even larger headroom, and that real programs only outperform controls in specific menu configurations, indicating that the apparent benefits may not be due to the actual layer computation choices.

By Yibei Guo, Rui Liu