arXiv AI

GrammarRL: Effective Grammar-Constrained Decoding via Reinforcement Learning

GrammarRL introduces a label‑free reinforcement learning approach that adapts language models to grammar constraints without annotated data. It optimizes two self‑supervised rewards—direct and reverse—using a Reinforce Leave‑One‑Out objective over grammar‑constrained rollouts, and regularizes toward a frozen base model. Experiments on sign‑language gloss translation, hierarchical text classification, and named entity recognition with Llama models show consistent gains over constrained greedy decoding and competitive performance to beam search while keeping inference cost low.

arXiv AI

4d ago

Stepwise Intrinsic Rewards for Reasoning in Large Language Models

The paper introduces Stepwise Marginal Information Gain (MIG), an intrinsic process reward that evaluates how each reasoning step of a large language model (LLM) or vision-language model (VLM) improves the likelihood of the reference answer. MIG rewards only new likelihood maxima, preventing duplicate credit, and is combined with outcome, format, and self‑distillation objectives to guide training. Experiments on eight benchmarks show that this method outperforms outcome‑only reinforcement learning and improves accuracy by up to 4.8 points over binary‑reward training, including a 12.6‑point gain on MathVerse and a 12.9‑point advantage on vision‑language transfer at 7B parameters.

By Xiangwei Wang, Wei Wang, Ken Chen, Nanduni Nimalsiri, Sachith Seneviratne, Saman Halgamuge

llms rag reinforcement-learning efficiency multimodal benchmarks

arXiv AI

Jun 2

Margin Adaptive DPO: Leveraging Reward Model for Granular Control in Preference Optimization

arXiv:2510. 05342v2 Announce Type: replace-cross Abstract: Direct Preference Optimization (DPO) has emerged as a simple and effective method for aligning large language models.

By Hyung Gyu Rho

llms nlp reinforcement-learning

arXiv AI

Jul 23

Co-Evolving LLM Evaluators and Policies via DynamicRubric

arXiv:2607. 20083v1 Announce Type: cross Abstract: Post-training with evaluator feedback on policy-induced samples serves as a major mechanism for improving large language models.

By Beining Wang, Weihang Su, Hongtao Tian, Hao Kong, Tao Yang, Ting Yao, Qingyi Pan, Yueyue Wu, Qingyao Ai, Min Zhang, Yiqun Liu

llms reinforcement-learning

arXiv AI

2d ago

Unlocking the Critic: Reward-Free Policy Optimization for LLM Post-Training

arXiv:2609.37119v1 Announce Type: cross Abstract: Recent approaches to reinforcement learning (RL) post-training for large language models increasingly remove the critic to reduce training instabilit...

By Hongyang Li, Xiao Li, Caesar Wu, Said Mammar, Gr\'egoire Danoy, Pascal Bouvry

llms reinforcement-learning safety

arXiv AI

Aug 25

SRPO: Self-Reflective Policy Optimization for Long-Horizon Reasoning

arXiv:2608.23493v1 Announce Type: new Abstract: Self-reflection is a powerful mechanism for credit assignment in human learning, converting sparse outcome feedback into actionable guidance. However,...

By Jialong Liu, Yuling Shi, Ning Yang, Xiaodong Gu, Zuchao Li

llms agents fine-tuning benchmarks

arXiv AI

Jun 30

The Hidden Cost of Structured Generation in LLMs: Draft-Conditioned Constrained Decoding

arXiv:2603. 03305v2 Announce Type: replace-cross Abstract: Large language models (LLMs) are increasingly used to generate executable outputs, JSON objects, and API calls, where a single syntax error can make the output unusable.

By Avinash Reddy, Thayne T. Walker, James S. Ide, Amrit Singh Bedi

llms benchmarks

arXiv AI

Jul 7

Safe Inference-Time Alignment via Lagrangian Reward Augmentation

arXiv:2607. 02781v1 Announce Type: cross Abstract: Inference-time alignment steers a frozen language model during decoding using auxiliary reward signals, avoiding the cost of repeated weight updates.

By Yaswanth Chittepu, Ativ Joshi, Sohini Chintala, Scott Niekum

llms rag reinforcement-learning fine-tuning safety

arXiv Machine Learning

Sep 10

Retrieval-augmented Decoding for Improving Truthfulness in Open-ended Generation

The paper introduces Retrieval-Augmented Decoding (RAD), a decoding-time method that improves the truthfulness of large language models without retraining. RAD uses a small reference set of up to ten annotated examples to build a grounding space of context embeddings and next-token logits, which it retrieves and aggregates during inference to shape the model’s output. Experiments on four open-ended generation benchmarks and four different LLMs show that RAD consistently outperforms strong baselines and generalizes well across tasks.

By Manh Nguyen, Sunil Gupta, Hung Le

llms rag reinforcement-learning fine-tuning benchmarks

arXiv AI

Aug 12

Evaluation-Conditioned Training: Teaching Models to Generalize to Stronger Oversight Regimes

arXiv:2608. 10209v1 Announce Type: new Abstract: Feedback signals used to train Large Language Models (LLMs) are the primary driver of their behavior and our main lever for instilling alignment with human values and objectives.

By Alec Harris, Kasey Corra, Archie Chaudhury, Yixiong Hao

llms reinforcement-learning fine-tuning safety

arXiv Machine Learning

Jul 21

Token-Level Off-Policy Learning for Faithful Generation Under Distribution Shift

arXiv:2607. 17524v1 Announce Type: cross Abstract: We propose Token-Level Off-Policy Labeling (TOPL), an off-policy training paradigm that reframes post-training as a token-level correctness prediction task.

By Zitong Huang, Gustavo Lucas Carvalho, Deqing Fu, Robin Jia

nlp fine-tuning benchmarks

arXiv AI

Jun 24

Matching Tasks to Objectives: Fine-Tuning and Prompt-Tuning Strategies for Encoder-Decoder Pre-trained Language Models

arXiv:2606. 24841v1 Announce Type: new Abstract: Prompt-based learning has emerged as a dominant paradigm in natural language processing.

By Ahmad Pouramini, Hesham Faili

llms nlp fine-tuning

arXiv Computation and Language

Sep 14

Text-to-SPARQL Generation with Reinforcement Learning: A GRPO-based Approach on DBLP

arXiv:2605.20066v2 Announce Type: replace Abstract: Knowledge graph question answering seeks to translate natural language questions into executable queries over knowledge graphs, but existing approa...

By Jann Pfeifer, Debayan Banerjee, Ricardo Usbeck

llms nlp reinforcement-learning fine-tuning benchmarks

Related stories