The paper introduces a two‑stage training framework for rubric‑based reinforcement learning (RL) that first uses rubric‑privileged on‑policy distillation (RP‑OPD) to provide dense token‑level supervision, then applies RL to optimize the rubric reward. In RP‑OPD, a student model learns to match a rubric‑aware teacher’s next‑token distributions on student‑generated prefixes. Experiments on HealthBench, ResearchQA, and RubricHub Science show that this two‑stage approach yields the highest scores and reduces reward hacking compared to a standard SFT + RL baseline.
arXiv:2608. 02948v1 Announce Type: cross Abstract: On-policy self-distillation (OPSD), where a single model acts as both student and teacher with different contexts, has shown promise in verifiable domains like math, where hard privileged information (PI) in the form of ground-truth answers structurally constrains valid continuations.
Post-training of reasoning language models is commonly driven by supervised distillation and reinforcement learning with verifiable rewards. Distillation often relies on chain-of-thought annotations that are expensive to obtain and may themselves be noisy, incomplete, or partially incorrect; even when the final solution is correct, an imperfect rationale can interfere with learning.
EvoRubric introduces a co‑evolutionary reinforcement learning framework that learns rubric‑driven evaluation criteria for open‑ended text generation. The system uses a shared policy that simultaneously generates responses and rubrics, while a frozen meta‑verifier and grader provide validity checks and scoring. Experiments on medical, writing, and science benchmarks show that EvoRubric outperforms static and other evolving‑rubric baselines, achieving higher average scores on five tasks.
arXiv:2607. 18082v1 Announce Type: cross Abstract: Rubric-based RL has recently shown promise in improving LLMs on open-ended tasks.
arXiv:2606. 19327v1 Announce Type: new Abstract: Post-training of reasoning language models is commonly driven by supervised distillation and reinforcement learning with verifiable rewards.
arXiv:2608. 09123v1 Announce Type: new Abstract: Aligning Large Language Models (LLMs) for open-ended tasks is challenging because responses must satisfy multidimensional criteria without following a single correct generation trajectory.
arXiv:2607. 18082v3 Announce Type: replace Abstract: Rubric-based RL has recently shown promise in improving LLMs on open-ended tasks.
The paper introduces ReVeR, a pointwise rubric reward model that derives hard rules and graded principles directly from instructions, avoiding response-induced bias. It employs tool-augmented judges and label‑blind distillation to create ReVeR‑Traces, a large dataset of rubric reasoning and judging trajectories. Experiments show ReVeR surpasses pairwise rubric models in preference accuracy, especially for instruction following, and reveal that higher preference accuracy does not always lead to better downstream policies.
The paper identifies a flaw in rubric‑based reinforcement learning where additive reward aggregation allows policies to compensate for missing critical criteria, leading to higher scores but poorer answers, especially in clinical consultation tasks. It demonstrates that grouping rubric criteria into protocol‑level dimensions—so a dimension only counts when all its criteria are satisfied—mitigates this reward hacking. The proposed Protocol‑level Rubrics (ProRubric) improve appropriateness by 10.8 points without sacrificing coverage and achieve the best performance across seven benchmarks.
arXiv:2606. 09887v1 Announce Type: cross Abstract: Reinforcement learning (RL) for large language models usually supervises reasoning with scalar outcome rewards, such as binary correctness.
RISE (Recursive Improvement via Self-Extrapolating Policy Distillation) is a new method that builds a synthetic teacher from a language model’s own RLVR training trajectory. By extrapolating the displacement between the current checkpoint and a trailing anchor in parameter or logit space, RISE transforms sparse outcome-based updates into dense token-level targets without external models or privileged conditioning. The approach recursively refines the student model, combining RLVR and on‑policy distillation, and demonstrates superior performance across mathematical reasoning, STEM, code generation, and multi‑turn agentic tasks.
arXiv:2607. 17247v1 Announce Type: cross Abstract: Large language model (LLM) post-training is essential for improving reasoning, adaptation, and alignment.