The paper investigates the optimization landscape of reinforcement learning with verifiable rewards (RLVR) on algorithmic tasks such as iterated group and quasigroup multiplication. By mapping entropy‑regularized RLVR with myopic tabular policies to a spin‑glass energy model, the authors prove that for a broad class of models with uncorrelated inputs the landscape is benign—there are no local minima that can trap training. They identify practical obstacles like diffusive barriers and gradient‑estimation errors, show these can be mitigated by choosing an appropriate entropy regulator, and demonstrate that a transformer trained from scratch with only last‑token rewards can learn an algorithmic chain of thought for iterated non‑Abelian group multiplications.
arXiv:2602. 14872v3 Announce Type: replace-cross Abstract: Reinforcement learning with verifiable rewards (RLVR) has been a main driver of recent breakthroughs in large reasoning models.
arXiv:2605. 17770v3 Announce Type: replace Abstract: The advancement of Large Reasoning Models (LRMs) has catalyzed a paradigm shift from reactive ``fast thinking'' text generation to systematic, step-by-step ``slow thinking'' reasoning, unlocking state-of-the-art performance in complex mathematical and logical tasks.
arXiv:2601. 15158v4 Announce Type: replace-cross Abstract: Transformers trained via Reinforcement Learning (RL) with outcome-based supervision can spontaneously develop the ability to generate intermediate reasoning steps (Chain-of-Thought).
arXiv:2602. 07832v3 Announce Type: replace-cross Abstract: Process rewards have been widely used in deep reinforcement learning to improve training efficiency, reduce variance, and prevent reward hacking.
arXiv:2511. 07317v2 Announce Type: replace-cross Abstract: We introduce Reinforcement Learning (RL) with Adaptive Verifiable Environments (RLVE), an approach using verifiable environments that procedurally generate problems and provide algorithmically verifiable rewards, to scale up RL for language models (LMs).
The paper investigates why reinforcement learning with verifiable rewards (RLVR) reduces the diversity of solutions in reasoning tasks. By analyzing the Countdown task, the authors show that RLVR contracts the solution space mainly at the entrance—before the first arithmetic operation—causing a 67% drop in solution coverage. They demonstrate that providing an unselected entrance prefix or applying entrance‑targeted interventions can restore or even improve coverage without harming accuracy.
arXiv:2507.14843v5 Announce Type: replace Abstract: Recent advances highlight Reinforcement Learning with Verifiable Rewards (RLVR) as a promising approach to enhancing LLM capabilities. However, it...
arXiv:2609.38598v1 Announce Type: new Abstract: Partially observable environments pose a fundamental challenge in deep reinforcement learning, requiring agents to compress temporal information from o...
arXiv:2606. 19771v1 Announce Type: new Abstract: Reinforcement Learning with Verifiable Rewards (RLVR) has significantly advanced Large Language Model (LLM) reasoning; however, it faces a fundamental optimization instability: uniform token updates precipitate entropy collapse, leading to premature convergence to suboptimal strategies, whereas excessive Shannon Entropy maximization can cause entropy explosion, driving blind exploration toward incoherent reasoning chains.
arXiv:2606. 17735v1 Announce Type: new Abstract: Although reinforcement learning (RL) has expanded the cognitive boundaries of large language models (LLMs), it often remains vulnerable to the autoregressive curse in long-horizon logical reasoning: small epistemic perturbations introduced early in generation can propagate irreversibly along the Markov decision process flow, triggering cascading failures that drive the reasoning trajectory toward collapse.
The paper examines the trade-off between using privileged expert distillation and standard reinforcement learning in partially observable settings. By analyzing a theoretical model called the perturbed Block MDP and running experiments on simulated locomotion tasks, it finds that the effectiveness of distillation depends on the stochasticity of latent dynamics and that the optimal latent policy is not always the best to distill. These insights provide guidelines for better leveraging privileged information to improve policy learning efficiency.
Rationale-Guided Policy Optimization (RGPO) is a reinforcement‑learning framework that adaptively uses ground‑truth rationale information to scaffold a language model’s reasoning process. Instead of treating reference solutions as fixed imitation targets, RGPO temporarily incorporates rationales to help the model generate better responses, then reverts to unguided learning with higher‑reward, model‑generated solutions. Experiments in both language‑only and vision‑language tasks show that RGPO consistently outperforms RLVR baselines, with ablation studies confirming that adaptive rationale guidance is a key factor in its success.