The article reports on a benchmark where the author evaluated TypeSafe AI’s Jev against large language models (LLMs) across 3,080 classification tasks, comparing metrics such as accuracy, latency, calibration, and confidence. It examines whether Jev can serve as a practical decision layer for AI systems, effectively moving AI from merely generating outputs to making informed decisions.
The article compares seven open decision models with a generative large language model (LLM) in a hands‑on repository called Jev. It evaluates each model based on routing accuracy, latency, and output reliability. The comparison provides practical insights into how these models perform in real‑world scenarios.
Can TypeSafe's Jev Make AI Agents Safer Without Another LLM? explores how a decision-first model can catch risky tool calls before they turn into real-world actions. The article discusses the potential of Jev to enhance AI agent safety by preventing unsafe actions without relying on an additional large language model.
The article titled "An Introduction to Jev" discusses an AI system that focuses on making decisions rather than generating text. It highlights Jev’s unique approach to decision-making within the broader context of AI development. The piece was originally published on Towards Data Science.
Exploring TypeSafe's Jev and comparing it with OpenAI on intent classification The post A New Kind of Model for AI Decision-Making? appeared first on Towards Data Science.
TypeSafe AI’s new model, Jev, is a ‘System One’ or decision model that takes text or semi‑structured data as input and outputs floating‑point probabilities for yes/no, choice, or score questions, rather than text. It charges only for input tokens ($0.042/million) and offers free output, making it cheaper and faster than typical LLMs. The API lets users build a state object (string, array, or key‑value pairs) and query it with multiple questions, receiving confidence scores and probability distributions for each answer.
The paper evaluates System One decision models—typed models that output probabilities for branching decisions—against supervised classifiers and generative language models on automated decision gate tasks. Eight checkpoints from six families, including the hosted model Jev, were benchmarked on workflow, intent, and social‑science items, showing that small trained classifiers match or slightly outperform decision models on intent and workflow when labels are available, while decision models outperform zero‑shot classifiers when labels are absent. The study also explores calibration, risk thresholds, and cost‑efficiency trade‑offs, providing condition‑dependent design guidelines for automated decision gates.
Understanding ow LLMs interact with the world around them, from returning data to taking action The post Tool Calling, Explained: How AI Agents Decide What to Do Next appeared first on Towards Data Science .
arXiv:2609.37647v1 Announce Type: cross Abstract: Jev is a commercial System One model from TypeSafe AI that does not generate text: given a state and typed questions, it returns a choice from fixed...
The study investigates whether Jev, a typed classifier that outputs probabilities over allowed answers without generating text, can replace large language model (LLM) rubric judges. Across nine panels from seven benchmarks, Jev’s accuracy differed significantly from LLM judges in only 8 of 27 paired comparisons, performing best on binary criteria and worse only on graded ones, while most other comparisons were inconclusive. In terms of cost and speed, Jev was 29 to 325 times cheaper and 30 to 220 times faster than the flash‑tier LLM judges, and a cascade approach that defers uncertain Jev verdicts to an LLM yielded only modest gains. whyItMatters":"The findings suggest that a lightweight classifier like Jev can serve as an efficient first‑stage evaluator, potentially reducing the reliance on expensive and slow LLM judges in automated grading pipelines."
System1-Arena is a benchmark that evaluates which decisions can be handled by System‑1 models—those that output probabilities in a single forward pass—versus System‑2 models that generate reasoning steps. It contains 92,000 questions across 105 tasks from 268 public sources, grouped into text classification, applied decisions, evaluation and control, and agent/action, and also includes 138 closed‑loop environments such as web shopping, coding control, robotics, games, and driving. The study tests 30 models, showing that large System‑1 models can match or exceed fast closed LLMs in non‑agentic accuracy with better‑calibrated probabilities at lower latency, while reasoning LLMs are most accurate but far more expensive, and large open LLMs perform best in closed‑loop settings.
The study compares Jev, a typed classifier that outputs probabilities over allowed answers, with three flash-tier LLM rubric judges across nine panels from seven benchmarks. Jev’s accuracy differs significantly from an LLM judge in only 8 of 27 paired comparisons, performing best on binary criteria and worse only on graded ones, while most other comparisons are inconclusive. In terms of cost and speed, Jev is 29 to 325 times cheaper and 30 to 220 times faster than the LLM judges, and a cascade that defers uncertain Jev verdicts to an LLM yields only modest gains of up to 2.0 points over the best single judge.
Multiple-Choice Benchmarks, Verifiers, Leaderboards, and LLM Judges with Code Examples
