The paper introduces Jev, a reinforcement‑learning‑trained model that provides calibrated probability answers to typed questions about a single input in one call. Jev is evaluated on RLCDAlignBench, a benchmark covering ten alignment failures across 44 tests and five target models, achieving a median AUROC of 0.886 zero‑shot and outperforming supervised baselines on most tasks. The study shows that question wording has little impact, while contextual fields that encode labels are more influential, and that Jev matches human‑label agreement while being 63× cheaper than LLM‑judge scorers.
By Ruoqi Guo, Yi Liu, Gelei Deng, Yuekang Li, Lida Zhao, Yutao Wu, Simin Chen, Ying Zhang, Leo Yu Zhang
arXiv:2609.36570v1 Announce Type: cross
Abstract: Indirect prompt injection makes an LLM agent treat untrusted retrieved text as instructions. We present CounterSteer, an inference-time defense that...
By Mark Russinovich
The paper evaluates TypeSafe AI’s Jev safety judge against GPT and GLM judges and local classifiers using a controlled adversarial test harness. Jev shows strong robustness on rule‑based tasks like input moderation, with minimal bypasses, but performs less well on judgment‑based tasks such as phishing detection, where it misses a higher proportion of phishing emails and is more prone to false positives. The study highlights Jev’s speed advantage and varying resilience depending on task type.
By Xiang Li, Aichun Huang, Mingming Zhang, Zheli Liu
This thesis introduces methods for assessing and enhancing the robustness of large language models (LLMs) against adversarial input variations. It proposes a generative robustness metric, R_stab(f), based on Jensen‑Shannon divergence, and proves bounds for localized attacks. The work also presents adaptive attacks, Trojan detection techniques, defense strategies for multi‑layer systems, and lightweight attestation for Model Context Protocol (MCP) agentic systems, all implemented in the JudgeGuard, TrojanArmor, and MCPSec suites.
By Narek Maloyan
arXiv:2609.38850v1 Announce Type: new
Abstract: Decision models such as Jev answer questions with probabilities, which are only useful if they are calibrated. Open-source reproductions rely on superv...
By Zhimin Gao, Pichao Wang
arXiv:2607. 05904v1 Announce Type: new Abstract: Training a language model against its own reference-free judgments (the premise of self-rewarding, self-play, and LLM-as-a-judge pipelines) assumes a model's verdict on a shown answer tracks correctness.
By Chenyu Zhou
JevAgentHarness is a new framework that benchmarks, deploys, and trains decision models to share control with large language models (LLMs) in agentic tasks. It introduces AgentDecisionBench to evaluate decision models from isolated questions to full agent control, revealing that accurate isolated decisions still miss many tasks and that naïve delegation can reduce success. By fitting confidence thresholds per choice and training a 4B decision model on states after agent mistakes, JevAgentHarness achieves high task success while cutting cost and speeding up execution, with results and resources made publicly available.
By Kunlun Zhu, Tao Feng, Xu Han, Heng Wang, Ling Yang, Xiangru Tang, Jiaxuan You, Heng Ji
arXiv:2609.37647v1 Announce Type: cross
Abstract: Jev is a commercial System One model from TypeSafe AI that does not generate text: given a state and typed questions, it returns a choice from fixed...
By Tobias Deu{\ss}er, Lorenz Sparrenberg, Rafet Sifa
arXiv:2610.16814v1 Announce Type: cross
Abstract: Large language models (LLMs) are increasingly used as judges to assign scalar scores that drive downstream decision-making, including tasks such as p...
By Anning Hu, Yangyu Li, Yuqi Huang, Shengchao Hu, Junchi Yan
arXiv:2607. 18966v1 Announce Type: new Abstract: Language models trained with reinforcement learning may learn to optimize the grader's judgment rather than the intended objective.
By Axel H{\o}jmark, J\'er\'emy Scheurer, Evgenia Nitishinskaya, Felix Hofst\"atter, Jason Wolfe, Theodore Ehrenborg, Bronson Schoen, Alexander Meinke
The paper surveys 58 studies on verifier errors in reinforcement learning with verifiable rewards (RLVR), examining how such errors affect training and which mitigations are effective. It finds that the type of error—rather than its frequency—has a larger impact on learning dynamics, with random errors mainly slowing progress and false positives potentially causing learning plateaus or collapse. The authors compile findings into comparison tables, reconcile conflicting reports, and propose a reporting protocol to standardize future research.
By Savyasachi Gupta, Aniruddha Srinivas Joshi
Large language model (LLM) agents now fix software, operate computers and serve customers, but their harnesses, the programs that run them, call a large LLM at every step, which makes long tasks slow...