arXiv:2608. 10812v1 Announce Type: cross Abstract: We study reference-free post-training for multilingual machine translation with open large language models.
Consensus Group Relative Policy Optimization (C‑GRPO) is a new training method that distills Minimum Bayes Risk (MBR) decoding into a group‑relative objective, enabling text generation models to approximate MBR performance without the costly inference‑time sampling and scoring. C‑GRPO only needs a utility function and policy samples, avoiding the need for gold references or curated preference data. Experiments on WMT 2024 machine translation and XSum summarization show that C‑GRPO matches MBR decoding quality while reducing inference overhead and outperforming other reference‑free baselines.
R3 is a new reward modeling framework that addresses limitations in current reward models by being rubric‑agnostic and generalizable across multiple evaluation dimensions. It provides interpretable, reasoned score assignments rather than opaque scalar outputs, enhancing transparency and flexibility in evaluating language models. The authors release their models, data, and code openly at https://github.com/rubricreward/r3.
arXiv:2606. 04807v1 Announce Type: new Abstract: Mitigating social bias in Large Language Models (LLMs) presents a distinct alignment challenge: unlike verifiable tasks, bias lacks a single ground truth, creating a high-variance, subjective reward landscape.
Mitigating social bias in Large Language Models (LLMs) presents a distinct alignment challenge: unlike verifiable tasks, bias lacks a single ground truth, creating a high-variance, subjective reward landscape. Previous preference-based fine-tuning methods have major trade-offs: Direct Preference Optimization (DPO) is limited by the lack of exploration inherent in offline training, while Proximal Policy Optimization (PPO) can lead to training instability due to potentially unreliable critic estimates.
GUPO: Gradient Uncertainty-aware Policy Optimization for Post-Training Large Language Models proposes a new method for fine‑tuning LLMs after training. The approach models each group gradient as a random variable, estimates its probability distribution, and uses Dirichlet‑based gradient uncertainty to weight each group’s contribution during policy updates. Experiments on multiple benchmarks show that this uncertainty‑aware aggregation improves the effectiveness of post‑training policy optimization.
The paper evaluates the scalability and adversarial generalization of Natural Language Inference (NLI) models trained with Group Relative Policy Optimization (GRPO) for Chain-of-Thought learning. By fine‑tuning 7B, 14B, and 32B language models with LoRA and QLoRA, the authors show strong performance on standard and adversarial NLI benchmarks, with the 32B model outperforming supervised baselines on adversarial sets. Using AWQ quantization, the 32B model fits within 22 GB of CUDA memory, demonstrating a scalable, practical framework for robust NLI without sacrificing inference quality.
The paper explores using reinforcement learning to enhance automatic text simplification for low‑resource languages, focusing on Catalan. It introduces a new reward function that blends the SARI metric with penalty terms, and applies Group Relative Policy Optimization (GRPO) to fine‑tune the IberianLLM‑7B‑Instruct model on the ASSET dataset. Post‑training, the model shows improved simplification performance on two Catalan benchmarks and reduces prior negative behaviors, though cross‑lingual transfer from English, Spanish, and Catalan translations of ASSET does not yield significant gains on an out‑of‑domain benchmark.
arXiv:2608. 13698v1 Announce Type: cross Abstract: Reinforcement Learning with Verifiable Rewards (RLVR), often optimized with Group Relative Policy Optimization (GRPO), has become a central recipe for improving the reasoning capabilities of pretrained language models but current studies remain heavily English-centric.
arXiv:2605. 21125v2 Announce Type: replace Abstract: Group Relative Policy Optimization (GRPO), a prominent algorithm within the Reinforcement Learning from Verifiable Rewards (RLVR) framework, has achieved strong results in improving the reasoning capabilities of large language models (LLMs).
The paper introduces Personalized Group Relative Policy Optimization (P‑GRPO), a new alignment framework for large language models that separates advantage estimation from immediate batch statistics. By normalizing advantages using preference‑group‑specific reward histories instead of the concurrent generation group, P‑GRPO maintains contrastive signals for distinct user preferences. Experiments across various tasks show that P‑GRPO converges faster and yields higher rewards than standard GRPO, improving alignment with heterogeneous human preferences while preserving general capabilities.
arXiv:2606. 05924v1 Announce Type: cross Abstract: Literary translation poses unique challenges due to the scarcity of high-quality annotated data and the need to balance expression fluency with literary effect.