arXiv AI

Just Ask Jev: Reinforcement Learning for Calibrated Decisions as a Zero-Shot Detector of AI Alignment Failures

The paper introduces Jev, a reinforcement‑learning‑trained model that provides calibrated probability answers to typed questions about a single input in one call. Jev is evaluated on RLCDAlignBench, a benchmark covering ten alignment failures across 44 tests and five target models, achieving a median AUROC of 0.886 zero‑shot and outperforming supervised baselines on most tasks. The study shows that question wording has little impact, while contextual fields that encode labels are more influential, and that Jev matches human‑label agreement while being 63× cheaper than LLM‑judge scorers.

arXiv AI

3d ago

JevAdvBench: A Benchmark and Black-Box Attacks for Reinforcement Learning for Calibrated Decisions Models

JevAdvBench introduces the first adversarial benchmark for reinforcement‑learning‑based calibrated decision (RLCD) models, providing 812 typed questions across 66 scenarios and a black‑box attack suite of 9,744 single‑edit variants. The benchmark evaluates attacks by comparing each perturbed decision to the model’s own clean decision and to an identical re‑run, revealing that rewording changes decisions by only 1.2 percentage points while certain injected opinions can flip 12.1% of decisions and lower confidence below 0.8 in 38% of cases. These findings demonstrate that RLCD models can be significantly misled by seemingly innocuous input edits, underscoring the need to treat the state as untrusted in applications.

By Jianyi Hu, Hangtao Zhang, Yi Liu, Yeqi Zeng, Li Zeng, Xianlong Wang, Rui Wang, Leo Yu Zhang

reinforcement-learning benchmarks safety

arXiv AI

1d ago

Evaluating and Benchmarking the System One Model Jev

arXiv:2609.37647v1 Announce Type: cross Abstract: Jev is a commercial System One model from TypeSafe AI that does not generate text: given a state and typed questions, it returns a choice from fixed...

By Tobias Deu{\ss}er, Lorenz Sparrenberg, Rafet Sifa

benchmarks

arXiv AI

Aug 10

Ask-E: An Environment for Calibrated Question Generation

arXiv:2608. 06933v1 Announce Type: cross Abstract: Today, we improve models by training and evaluating them on problems at the frontier of their abilities.

By Sarah Pratt, Jae Sung Park, Scott Geng, Ali Farhadi

llms benchmarks

arXiv Machine Learning

Aug 19

Thinking in a Low-Resource Language: What SFT Builds, What RL Fixes, What Accuracy Cannot See

The paper evaluates how three large mixture‑of‑experts models (Alibaba, OpenAI, NVIDIA) can be fine‑tuned to reason in a low‑resource language, specifically Greek. Accuracy metrics show little change, but the authors uncover significant qualitative improvements: after supervised fine‑tuning, models reason in Greek on ~98% of items, with better grammaticality and retained general ability. Reinforcement learning with pre‑registered rewards further eliminates reasoning‑channel leaks and format skips, while the Greek‑reasoning habit remains robust to an accuracy‑only gradient.

By Ayoub Kirouane, Christos Petrocheilos

reinforcement-learning fine-tuning benchmarks

arXiv AI

Aug 18

Measuring Reward Hacking and Reasoning-Answer Decoupling Under Position-Confounded Optimization

arXiv:2608. 15445v1 Announce Type: new Abstract: When a reward is correct on every training example yet consistent with more than one goal, a model can acquire an unintended one, a failure known as goal misgeneralization.

By Suyash Maniyar, Armaan Sandhu, Abhishek Mishra

llms benchmarks

arXiv Machine Learning

Jun 9

TinyJudge: Unverifiable Constraint Alignment via Lightweight Specialist Ensembles

arXiv:2606. 07520v1 Announce Type: cross Abstract: Instruction Following (IF) is a core capability of LLMs, requiring strict adherence to diverse constraints, ranging from verifiable ones (e.

By Yirong Zeng, Yufei Liu, Xiao Ding, Yutai Hou, Yuxian Wang, Wu Ning, Haonan Song, Dandan Tu, Qixun Zhang, Yuxiang He, Bibo Cai, Ting Liu

llms reinforcement-learning benchmarks safety

arXiv AI

Jul 7

When Rubrics Fail: Error Enumeration as Reward in Reference-Free RL Post-Training for Virtual Try-On

arXiv:2603. 05659v3 Announce Type: replace-cross Abstract: Reinforcement learning with verifiable rewards (RLVR) and Rubrics as Rewards (RaR) have driven strong gains in domains with clear correctness signals and even in subjective domains by synthesizing evaluation criteria from ideal reference answers.

By Wisdom Ikezogwo, Mehmet Saygin Seyfioglu, Ranjay Krishna, Karim Bouyarmane

reinforcement-learning benchmarks

arXiv AI

Aug 7

OSReward: Instituting Standardized Evaluation for Cross-Platform Computer-Use Reward Models

arXiv:2607. 28609v2 Announce Type: replace Abstract: Computer-using agents (CUAs) are advancing rapidly across the digital world.

By Qiushi Sun, Kanzhi Cheng, Yian Wang, Bowen Yang, Hang Yan, Liheng Chen, Fangzhi Xu, Zichen Ding, Nuo Chen, Jialin Cao, Xingdong Gong, Zehao Li, Kaiming Jin, Xinfeng Yuan, Zhoumianze Liu, Jingyang Gong, Zhangyue Yin, Jiahui Gao, Zhiyong Wu, Tianbao Xie, Jianbing Zhang, Ben Kao, Lingpeng Kong

llms agents reinforcement-learning multimodal benchmarks safety

arXiv Computation and Language

Sep 1

Small Language Models as Judges for Rubric-Based Reinforcement Learning

arXiv:2608.30005v1 Announce Type: new Abstract: Rubric-based reinforcement learning extends RL beyond tasks with exact answers or rule-based verifiers by scoring responses against instance-specific c...

By Fengyu Xie, Yilun Zhao, Bingsen Chen, Arman Cohan, Chen Zhao

llms reinforcement-learning

arXiv AI

Jul 3

Scaling with Confidence: Calibrating Confidence of LLMs for Adaptive Test Time Scaling

arXiv:2607. 01612v1 Announce Type: new Abstract: Training large language models (LLMs) with reinforcement learning (RL) has significantly advanced their performance on reasoning and question-answering tasks.

By Xuqing Yang, Yi Yuan, Shanzhe Lei, Xuhong Wang

llms reinforcement-learning multimodal benchmarks

arXiv Machine Learning

Jun 5

Alignment Risks from Capability-Seeking RL Training

arXiv:2602. 12124v2 Announce Type: replace Abstract: While most AI alignment research focuses on preventing models from generating explicitly harmful content, a more subtle risk arises from capability-seeking RL training in vulnerable environments.

By Yujun Zhou, Yue Huang, Han Bao, Kehan Guo, Zhenwen Liang, Pin-Yu Chen, Tian Gao, Werner Geyer, Nuno Moniz, Nitesh V Chawla, Xiangliang Zhang

llms reinforcement-learning fine-tuning safety

arXiv AI

Sep 23

When and How Should an Agent Clarify? CIGAsk: Teaching LLMs to Clarify via Counterfactual Information Gain

arXiv:2609.24290v1 Announce Type: new Abstract: Instruction-tuned LLMs faced with underspecified queries often commit to a single interpretation rather than ask for clarification, producing confident...

By Yunxiang Li, Xixin Wu, Helen Meng

llms agents reinforcement-learning benchmarks

Related stories