Given a state and a set of options, such as intents, tools or interface elements, System One decision models return a distribution over the options in one forward pass, without generated reasoning. We...
Prism-1 introduces Position- and Rival-Independent Scoring for System One models, ensuring that option ordering and set changes do not affect decision probabilities. It achieves this by reading each option in its own block at shared positions, preserving softmax independence across options. Experiments on Qwen3-8B and Gemma 4 12B show Prism-1 maintains baseline accuracy while eliminating sensitivity to shuffling and irrelevant options.
The paper investigates how compact an encoder can be for closed‑set decision tasks in language‑model pipelines, comparing joint versus per‑option scoring. In a 32‑million‑parameter study, per‑option scoring outperforms joint scoring by 5.8 chance‑corrected skill points on a 37‑benchmark suite, and two external benchmarks corroborate this advantage. Across six released joint encoders ranging from 17 million to 1 billion parameters, the 68‑ and 150‑million‑parameter models surpass a 435‑million‑parameter DeBERTa‑v3 zero‑shot classifier on 209 overlapping tasks and match it on 145 unseen tasks, while the smallest model processes a median request in 14 ms versus 728 ms for the classifier.
Prefill‑only decision models evaluate every candidate in a menu in a single forward pass, avoiding decoding and reducing cost by one to two orders of magnitude compared to generative language models. The paper demonstrates that when only the candidate menu changes, the model’s post‑intervention accuracy can be predicted solely from the cached first‑pass distribution using a simple estimator that renormalizes and selects the argmax, without any labels or second pass. Across seven model families, ten datasets, and two task types, this menu‑only intervention prediction is within 4.2 points of actual accuracy, and in one family it is exact, whereas a probability‑level variant fails by 21 points, indicating the property resides in ranking rather than calibrated probabilities.
LEV is a decision model that performs reasoning by looping its layers instead of generating text, allowing it to deepen its computation on a per-question basis. Trained jointly across all loop steps, a single LEV checkpoint can answer at any depth, improving accuracy—especially on hard questions—while keeping inference fast. On benchmark tests, LEV outperforms dense and cascade models by 5.4 to 8.8 accuracy points with sub‑50 ms latency, and on‑demand depth selection saves 15–24% of computation for minimal accuracy loss.
arXiv:2610.07730v1 Announce Type: cross Abstract: Typed decision models answer a declared question without generating text: a decision head returns a probability for each of the declared options in a...
arXiv:2608. 11947v1 Announce Type: cross Abstract: Multiple-choice benchmarks are widely used to evaluate large language models, but MCQ scores conflate knowledge with sensitivity to option order, which makes them unreliable measures of model knowledge.
The paper investigates whether accuracy and order sensitivity in large language model (LLM) multiple‑choice question (MCQ) evaluation move together. Using proprietary and open‑weight models on MMLU and ARC‑Challenge, the authors measure order sensitivity with flip rate and recall standard deviation (RStd) and find that these metrics often diverge from accuracy and from each other. They conclude that accuracy alone is insufficient and recommend reporting both flip rate and RStd alongside accuracy.
Verdict-MQ is an open encoder that processes a single piece of state once and then answers multiple typed questions about that state, avoiding separate passes for each question. By using a structured attention mask, it ensures the state encoding is independent of question order and prevents questions from influencing one another. On the typed‑decisions benchmark, Verdict‑MQ outperforms a larger specialist model, achieving higher accuracy and better calibration, while on other benchmarks it shows mixed results compared to fine‑tuned baselines.
arXiv:2607.27422v2 Announce Type: replace Abstract: Goal-conditioned action generators can produce many plausible actions. Best-of-\(K\) selection then uses a critic to rank those actions for the cur...
The paper investigates how the order of candidate documents in a prompt affects the decisions made by large‑language‑model (LLM) scorers, even when their ranking quality is similar. It shows that five scorers with only a 0.010 nDCG@10 difference can produce retained‑set overlaps as low as 0.66–0.84, and that existing rerankers still exhibit significant order dependence. The authors propose Order‑Consistency SFT (OC‑SFT), a training method that reduces this dependence, maintaining ranking quality while improving decision stability across multiple tasks.
arXiv:2610.02267v1 Announce Type: new Abstract: Agent harnesses make many small, typed decisions per task: which model to call, which tool to use, whether retrieved text is relevant, whether an input...