The paper introduces Token Importance-Aware Policy Optimization (TIAO), a reinforcement learning approach that improves text summarization by weighting token importance based on token dependency. TIAO reweights a trajectory’s advantage according to the overall dependencies of core tokens, addressing the limitation of previous methods that treat all tokens equally. Experiments demonstrate that a 7B foundation model enhanced with TIAO achieves performance comparable to GPT‑4 and GPT‑5‑nano on real‑world datasets.
arXiv:2607. 17524v1 Announce Type: cross Abstract: We propose Token-Level Off-Policy Labeling (TOPL), an off-policy training paradigm that reframes post-training as a token-level correctness prediction task.
We propose Token-Level Off-Policy Labeling (TOPL), an off-policy training paradigm that reframes post-training as a token-level correctness prediction task. Our key intuition is that by training the model to distinguish good and bad tokens in a response, we naturally guide the model towards generating good tokens, while avoiding the pitfalls that come with directly training the model to generate off-policy tokens.
The paper introduces PLUS, a framework that uses reinforcement learning to generate text-based summaries of individual users’ preferences, characteristics, and past conversations. These summaries condition a reward model, allowing it to predict personalized response preferences and improving reward accuracy by 11–77 % over the standard Bradley‑Terry model. PLUS demonstrates robust performance with new users and topics, achieves a 25 % improvement over existing personalized RLHF techniques, and enables zero‑shot personalization for state‑of‑the‑art models like GPT‑4.
arXiv:2608. 09605v1 Announce Type: cross Abstract: Large Language Models (LLMs) have emerged as powerful tools for improving recommendation systems.
arXiv:2602. 13940v2 Announce Type: replace-cross Abstract: Tokenization is a hardcoded compression step which remains in the training pipeline of Large Language Models (LLMs), despite a general trend towards architectures becoming increasingly end-to-end.
arXiv:2607. 04728v1 Announce Type: cross Abstract: Reinforcement learning (RL) post-training for large language models (LLMs) follows a efficient paradigm of "rollout then update", which inevitably results in off-policy training data.
Consensus Group Relative Policy Optimization (C‑GRPO) is a new training method that distills Minimum Bayes Risk (MBR) decoding into a group‑relative objective, enabling text generation models to approximate MBR performance without the costly inference‑time sampling and scoring. C‑GRPO only needs a utility function and policy samples, avoiding the need for gold references or curated preference data. Experiments on WMT 2024 machine translation and XSum summarization show that C‑GRPO matches MBR decoding quality while reducing inference overhead and outperforming other reference‑free baselines.
Reinforcement learning (RL) post-training for large language models (LLMs) follows a efficient paradigm of "rollout then update", which inevitably results in off-policy training data. To resolve this, Importance sampling (IS) is proposed, while the token-level ratios compound over long sequences, causing severe variance exploded.
arXiv:2608.30632v1 Announce Type: cross Abstract: Reinforcement learning (RL), particularly RL with Verifiable Rewards (RLVR), has recently emerged as a central paradigm for enhancing large language...
arXiv:2602. 10238v2 Announce Type: replace-cross Abstract: The growing size of Large Language Models (LLMs) makes efficient inference challenging, primarily due to the memory demands of the autoregressive Key-Value (KV) cache.
arXiv:2602. 19327v3 Announce Type: replace Abstract: A significant portion of recent research on Large Language Model (LLM) alignment focuses on developing new policy optimization methods based on Group Relative Policy Optimization (GRPO).
arXiv:2606. 27550v1 Announce Type: cross Abstract: Multi-token prediction has been shown to increase data density during training, improve downstream text-generation quality, and serves as the defacto approach for self-speculative decoding.