The paper introduces Solver-Gradient Guided Reinforcement Learning (SG‑RL), a method that augments standard RL with bounded gradients from a differentiable MPC solver to adapt cost‑function weights online. SG‑RL integrates solver‑gradient guidance into PPO through actor‑update scaling, policy loss, advantage estimation, and value‑function learning, achieving comparable or superior closed‑loop performance while requiring up to 70.6% fewer samples. Experiments on two autonomous racing platforms with intentional model mismatch demonstrate that SG‑RL outperforms both RL and gradient‑based policy learning baselines and generalizes zero‑shot to unseen environments.
arXiv:2605. 04568v3 Announce Type: replace-cross Abstract: State-of-the-art model-based Reinforcement Learning (RL) approaches either use gradient-free, population-based methods for planning, learned policy networks, or a combination of policy networks and planning.
The paper introduces Multi‑Fidelity Policy Gradient (MFPG) methods that combine scarce, high‑fidelity (HF) reinforcement‑learning data with abundant, low‑fidelity (LF) data to stabilize training. Unlike prior approaches that directly optimize biased LF objectives, MFPG uses LF data only to construct a control variate that reduces variance without biasing the policy gradient. The authors extend MFPG to modern actor‑critic algorithms, notably a budget‑aware MFPG‑PPO that preserves cross‑fidelity correlation, monitors estimator uncertainty, and allocates a fixed sampling budget between HF and LF sources. Experiments on simulated robot locomotion and a physical Franka arm show that MFPG‑PPO consistently outperforms standard PPO trained on HF data alone and matches performance achieved with 16× more HF data on the hardest tasks, while enabling stable learning with as few as four real‑robot episodes per update.
arXiv:2607. 18830v1 Announce Type: cross Abstract: Model-Agnostic Meta-Learning (MAML) is a widely used framework for reinforcement learning (RL) that enables efficient transfer by learning global policy parameters that can be rapidly adapted to new tasks.
arXiv:2510. 17709v2 Announce Type: replace-cross Abstract: Training Reinforcement Learning (RL) policies using simulation models before deployment in real-world environments is a common strategy when real-world interaction is expensive.
arXiv:2605. 20256v2 Announce Type: replace Abstract: Reinforcement learning has become a cornerstone for aligning and unlocking the reasoning capabilities of large-scale models.
arXiv:2608. 02951v1 Announce Type: cross Abstract: Preference-based reinforcement learning (PbRL) for general stochastic MDPs often requires training a reward model.
arXiv:2606. 08779v1 Announce Type: new Abstract: Reinforcement Learning (RL) has emerged as a pivotal post-training paradigm, yet it frequently suffers from unpredictable sub-optimum performance or even training collapses.
Agentic ESOpt proposes using evolution strategies (ES) instead of reinforcement learning to fine‑tune large language‑model agents for long‑horizon tasks. ES offers model scalability, flexibility, and better long‑horizon credit assignment, enabling full‑parameter optimization with minimal GPU memory. The framework samples parameter perturbations, evaluates agents with rewards, and updates online, achieving notable performance gains on WebArena‑Lite and in test‑time prompt‑parameter co‑evolution.
BCPPO is a new variant of Proximal Policy Optimization that uses Bachelier-inspired cost‑prediction networks to generate a smooth penalty based on disagreement among critics. The method keeps temporal‑difference learning unchanged, applies a saturation‑aware controller to manage cost penalties, and deploys only the policy network. Across extensive experiments, BCPPO outperforms comparators in achieving higher mean returns while maintaining lower or comparable CVaR in all tested tasks.
arXiv:2601. 00898v3 Announce Type: replace Abstract: Diffusion-based policies have gained growing popularity in solving a wide range of decision-making tasks due to their superior expressiveness and controllable generation during inference.
The paper introduces Trajectory-Aware Policy Optimization (TAPO), a new framework for reinforcement learning that addresses the state distribution shift caused by long-horizon autoregressive rollouts in large language models. TAPO derives a state-distribution correction from the off-policy policy gradient, integrates it into the RL objective, and uses a geometric-mean estimator for variance reduction along with length-aware dynamic clipping to stabilize updates. Experiments on eight competition-level benchmarks show TAPO outperforms existing LLM RL methods, achieving up to 48% improvement on mathematical reasoning and 12% on algorithmic coding tasks while enhancing training stability.
arXiv:2606. 11087v1 Announce Type: cross Abstract: Expressive continuous control policies, such as diffusion and flow models, form the backbone of recent advances in scaling imitation learning for simulated and real robot control.