arXiv Computation and Language
RubriX: A Fast, Fine-Tunable Non-Autoregressive Rubric Judge
Read the original on arXiv Computation and Language →The Flow has not summarised this story yet — read it at arXiv Computation and Language.
Related stories
arXiv AI
Jun 9
SLMJury: Can Small Language Models Judge as Well as Large Ones?
arXiv:2606. 07810v1 Announce Type: cross Abstract: Large language models (LLMs) are widely used as judges for evaluating model outputs, but their high cost, latency, and opacity limit scalability.
arXiv AI
Sep 30
Evaluating and Benchmarking the System One Model Jev
arXiv:2609.37647v1 Announce Type: cross Abstract: Jev is a commercial System One model from TypeSafe AI that does not generate text: given a state and typed questions, it returns a choice from fixed...
arXiv Machine Learning
Oct 9
Evaluating Rubric Generation with Interventional Transfer
The paper introduces Interventional Transfer (IT), a method for evaluating rubric generation by assessing how two rubrics respond to perturbations that cause a pass/fail shift. IT reveals asymmetries in rubrics generated by different LLMs when applied to HealthBench, showing that improvements on one rubric do not reliably transfer to the other. The study contrasts IT with existing evaluation approaches, highlighting its ability to uncover these asymmetries.
arXiv AI
Oct 31
Rubricon: Automated Rubric Generation and Refinement for LLM Evaluation and Training
Rubricon is a pipeline that automatically generates and refines discriminative rubrics for evaluating large language models (LLMs) without requiring human preference labels or manual rubric creation. It samples multiple rubrics, selects the most discriminative ones, adds missing criteria, prunes misaligned or shallow criteria, and uses LLM self‑critique and observed pass rates to guide refinement. Across diverse LLM benchmarks, Rubricon’s rubrics achieve 9% better alignment with expert‑authored rubrics than other synthetic methods, and training a Qwen3.5‑9B model on HealthBench with Rubricon rubrics improves performance by 4.5% over the best alternative synthetic generator.
arXiv Computation and Language
Sep 1
Small Language Models as Judges for Rubric-Based Reinforcement Learning
arXiv:2608.30005v1 Announce Type: new Abstract: Rubric-based reinforcement learning extends RL beyond tasks with exact answers or rule-based verifiers by scoring responses against instance-specific c...
arXiv Computation and Language
Sep 3
Can LLM-as-a-Judge Reliably Verify Rubrics in Agentic Scenarios?
The paper introduces RuVerBench, a benchmark with 2,458 instances for evaluating the reliability of Large Language Models acting as judges (LaaJ) in verifying rubric compliance within agentic scenarios such as deep research and agentic coding. It systematically meta‑evaluates frontier LLMs, revealing that even the most advanced models perform well yet still produce substantial noise. The study also examines how prompt design, batching, and majority voting affect verification accuracy, noting that weaker models are more prompt‑sensitive, batched verification trades accuracy for efficiency, and majority voting offers diminishing returns.