arXiv Computation and Language
GenRubric: Self-Evolving Rubric Generation for Scalable LLM Evaluation
Read the original on arXiv Computation and Language →The Flow has not summarised this story yet — read it at arXiv Computation and Language.
Related stories
arXiv AI
Jul 23
Co-Evolving LLM Evaluators and Policies via DynamicRubric
arXiv:2607. 20083v1 Announce Type: cross Abstract: Post-training with evaluator feedback on policy-induced samples serves as a major mechanism for improving large language models.
arXiv AI
Sep 4
Judging LLM-as-a-Judge: Concerning Rubric Artifacts in LLM-based Automated Text Generation Evaluation
The paper examines LLM-as-a-Judge systems used to assess AI-generated text, questioning the assumption that judgments are derived from reasoning over responses and rubrics. It finds that classifiers trained solely on rubric text can predict judge outputs, indicating that rubrics contain recoverable evaluative signals independent of the responses. Counterfactual experiments show judges often fail to adjust decisions when either the response or rubric criterion is reversed, raising doubts about the reliability of rubric-based LLM evaluation.
arXiv Computation and Language
Oct 12
From Structure to Validity: Mechanism-Guided Generation and Refinement of LLM Rubrics
The paper investigates what makes rubric design effective for evaluating language models, analyzing criterion organization, operationalization, and score aggregation. It finds that a well-structured organization supports comprehensive criteria, semantic specification alone is insufficient for accurate judgments, and complex aggregation yields limited gains. Based on these insights, the authors propose ARTS‑Hybrid, an adaptive tree search with hybrid refinement that improves both evaluation performance and rubric quality, and shows promise for guiding model optimization.
Hugging Face Trending Papers
Jun 3
Self-Evolving Deep Research via Joint Generation and Evaluation
Large Language Models (LLMs) have become increasingly adopted in daily applications, with deep research standing out as a particularly important capability. Unlike traditional question-answering (QA) tasks, deep research report generation lacks definitive ground-truth, making reward design inherently unverifiable and limiting effective reinforcement learning.
arXiv Machine Learning
Oct 9
Evaluating Rubric Generation with Interventional Transfer
The paper introduces Interventional Transfer (IT), a method for evaluating rubric generation by assessing how two rubrics respond to perturbations that cause a pass/fail shift. IT reveals asymmetries in rubrics generated by different LLMs when applied to HealthBench, showing that improvements on one rubric do not reliably transfer to the other. The study contrasts IT with existing evaluation approaches, highlighting its ability to uncover these asymmetries.
arXiv Computation and Language
Oct 9
EvoRubric: Self-Evolving Rubric-Driven RL for Open-Ended Generation
EvoRubric introduces a co‑evolutionary reinforcement learning framework that learns rubric‑driven evaluation criteria for open‑ended text generation. The system uses a shared policy that simultaneously generates responses and rubrics, while a frozen meta‑verifier and grader provide validity checks and scoring. Experiments on medical, writing, and science benchmarks show that EvoRubric outperforms static and other evolving‑rubric baselines, achieving higher average scores on five tasks.