arXiv AI October 26, 2026 By Hosein Hasanbeig, Priyanshu Gupta, Ashish Tiwari, Vu Le, Sumit Gulwani

RubricCraft: Where Solutions Diverge, Criteria Emerge

RubricCraft is a method that generates natural‑language evaluation rubrics for open‑ended, artifact‑producing language tasks by analyzing differences among independently sampled solver trajectories. It constructs rubric criteria based on how each attempt satisfies or violates task requirements, then applies and refines these criteria to better detect errors and recognize partial correctness without needing a single reference solution. On the SpreadsheetBench v2 benchmark, RubricCraft’s rubrics reduce the proportion of incorrect outputs that pass (29.5% vs. 42.6%) and improve ranking of outputs while maintaining consistency with the benchmark’s own checker.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

arXiv Machine Learning

Oct 9

Evaluating Rubric Generation with Interventional Transfer

The paper introduces Interventional Transfer (IT), a method for evaluating rubric generation by assessing how two rubrics respond to perturbations that cause a pass/fail shift. IT reveals asymmetries in rubrics generated by different LLMs when applied to HealthBench, showing that improvements on one rubric do not reliably transfer to the other. The study contrasts IT with existing evaluation approaches, highlighting its ability to uncover these asymmetries.

By Erik Skalnes, Layne C. Price, Raviteja Anantha, Michael Oberst

llms benchmarks

arXiv Computation and Language

Sep 3

Can LLM-as-a-Judge Reliably Verify Rubrics in Agentic Scenarios?

The paper introduces RuVerBench, a benchmark with 2,458 instances for evaluating the reliability of Large Language Models acting as judges (LaaJ) in verifying rubric compliance within agentic scenarios such as deep research and agentic coding. It systematically meta‑evaluates frontier LLMs, revealing that even the most advanced models perform well yet still produce substantial noise. The study also examines how prompt design, batching, and majority voting affect verification accuracy, noting that weaker models are more prompt‑sensitive, batched verification trades accuracy for efficiency, and majority voting offers diminishing returns.

By Yangda Peng, Yunjia Qi, Haotian Xia, Guanzhong He, Xintong Shi, Richeng Xuan, Songyuanyi Lu, Yixian Liu, Zhichao Hu, Yuhong Liu, Hao Peng

llms agents benchmarks

arXiv AI

Jul 7

When Rubrics Fail: Error Enumeration as Reward in Reference-Free RL Post-Training for Virtual Try-On

arXiv:2603. 05659v3 Announce Type: replace-cross Abstract: Reinforcement learning with verifiable rewards (RLVR) and Rubrics as Rewards (RaR) have driven strong gains in domains with clear correctness signals and even in subjective domains by synthesizing evaluation criteria from ideal reference answers.

By Wisdom Ikezogwo, Mehmet Saygin Seyfioglu, Ranjay Krishna, Karim Bouyarmane

reinforcement-learning benchmarks

arXiv AI

Oct 15

Instructions Are Evidence, Not Objectives: Measuring Outcome Alignment of Agents

arXiv:2610.17757v1 Announce Type: new Abstract: Large language model agents can follow complex human instructions to solve real-world tasks. In practice, however, initial instructions are often incom...

By Kaiming Liu, Yuxuan Liu, Jinrui Ju, Minyuan Ruan, Xuanyu Lei, Peng Li, Yang Liu

llms agents benchmarks safety

arXiv Computation and Language

Oct 15

Interlock: From Local Correctness to Task Completion in Long-Horizon Agents

Interlock is a framework that transforms long‑horizon tasks into typed completion graphs, linking evidence, decisions, and required effects through grounding, coupling, and realization edges. It compiles these graphs into a resettable workspace, a public request, and an evaluation rubric, and uses source‑coverage, public‑input, and agent‑as‑judge reviews to ensure that all mandatory requirements are met. Fine‑tuning Qwen3.6 backbones on Interlock‑generated trajectories improves pass rates on AutomationBench, JobBench, and Harvey LAB, showing that the synthesized experience benefits external task environments.

By Linji Hao, Yanji He, Zihan Wang, Hansi Yang, Jiayuan Zhang, Hongyan Hao, Qi Gu, Yan Feng, Han Wu

llms agents fine-tuning

Hugging Face Trending Papers

Oct 14

SoK: Grading Open-Text Legal Answers, from Law School Exams to Trustworthy Rubric Judges

Law students are examined through open-text answers: case solutions, essays and memos that apply the law to a set of facts. Such answers are expensive to grade, human graders disagree widely on them,...