← Back to all news
arXiv AI

PPO-HRAP: Proximal Policy Optimization with a Hybrid Regime-Aware Policy for Risk-Controlled Trading

PPO‑HRAP introduces a hybrid regime‑aware policy that blends Proximal Policy Optimization with a volatility‑conditioned regime prior to balance upside participation and drawdown control in trading. The agent uses market and portfolio features, rewards that combine log return, VIX‑conditioned drawdown penalty, exposure deviation, and turnover cost, and outputs a blended action between the PPO actor and the regime‑derived target exposure. In backtests on SPY (2020‑2022) it achieved a 27.62% total return, 8.48% annualized return, and reduced maximum drawdown from 34.10% to 18.47%, while maintaining stable performance across multiple seeds and ranking first on total return and Sharpe ratio in single‑run cross‑asset tests on QQQ and DIA.

Related stories

arXiv AI
Jun 9

TT-DAC-PS: Twin-Target Deterministic Actor-Critic with Policy Smoothing for Optimal Trade Execution

arXiv:2606. 08379v1 Announce Type: new Abstract: This study addresses the optimal execution of large stock sell programs by introducing TT-DAC-PS (Twin-Target Deterministic Actor-Critic with Policy Smoothing), a deterministic actor-critic architecture that combines twin exponential-moving-average critic targets with pessimistic min backup, TD3-style target policy smoothing noise, delayed actor updates, and conservative Q regularisation to curb overestimation.

By Ilia Zaznov, Atta Badii, Julian Kunkel, Alfonso Dufour
reinforcement-learningbenchmarks
More like this →
arXiv Machine Learning
Jul 20

CLaC@FinMMEval 2026 Task 3: Sentiment-Augmented Deep Reinforcement Learning for Active Trading -- An Alpha-Reward Approach

arXiv:2607. 16028v1 Announce Type: new Abstract: This paper presents our system for Task 3 of the CLEF 2026 FinMMEval Lab, which requires daily long, flat, or short trading decisions for Bitcoin (BTC) and Tesla (TSLA) using news and historical market data.

By Andrei Neagu, Eeham Khan, Leila Kosseim
llmsagentsreinforcement-learning
More like this →
arXiv Machine Learning
Aug 6

Adaptive Finite-Budget Training for CVaR Risk-Aware Q-Learning

arXiv:2608. 04305v1 Announce Type: new Abstract: Risk-aware Q-learning (RaQL) provides a model-free, two-timescale estimator for dynamic risk objectives, but its finite-budget behavior remains fragile: fixed inner-loop hyperparameters can produce unstable value estimates, persistent Bellman residuals, and inefficient sample reuse.

By Yifan Wu, Junjie Lei, Wenjie Huang
reinforcement-learning
More like this →
arXiv AI
Jun 9

GIFT: LLM-Guided State-Reward Interface for Financial Reinforcement Learning

arXiv:2606. 08450v1 Announce Type: new Abstract: Financial portfolio trading is naturally formulated as a reinforcement learning problem, where an agent sequentially rebalances assets under changing market conditions to balance return, risk, and transaction costs.

By Yanyan Wu, Boyi Zhang, Yanlin Liu, Xinyu Fang, Jining Luan, Meiqi Zhang, Jiacheng Liu, Hao Zeng, Dexu Yu, Chang Liu, Hanwen Du, Yongxin Ni, Youhua Li
llmsagentsreinforcement-learning
More like this →
arXiv AI
Sep 1

BCPPO: Bachelier-Inspired Constrained Proximal Policy Optimization for Tail-Risk-Aware Safe Reinforcement Learning

BCPPO is a new variant of Proximal Policy Optimization that uses Bachelier-inspired cost‑prediction networks to generate a smooth penalty based on disagreement among critics. The method keeps temporal‑difference learning unchanged, applies a saturation‑aware controller to manage cost penalties, and deploys only the policy network. Across extensive experiments, BCPPO outperforms comparators in achieving higher mean returns while maintaining lower or comparable CVaR in all tested tasks.

By Dongsheng Hou, Yanqiao Chen, Yuhan Rui
reinforcement-learningsafety
More like this →
arXiv Machine Learning
Aug 28

Shared Actors Need Not Share Critics: Effects of Value Mismatch in Parallel Reinforcement Learning

The paper investigates the problem of sharing a single critic across multiple parallel environments in reinforcement learning. It shows that when environments assign different expected returns to the same state, a shared critic must reconcile conflicting value targets, which can distort advantage estimates and misguide policy updates. The authors propose a simple fix—providing the critic with the environment index—demonstrating through bandit models and experiments on CartPole, MuJoCo, BipedalWalker, and 16 Procgen games that this conditional critic stabilizes learning and boosts returns, achieving a 40.8% improvement in aggregate normalized return on unseen levels.

By Zhenya Liu, Yang Meng, Zhuokai Zhao, Xuefeng Liu, Yuxin Chen
reinforcement-learning
More like this →
arXiv AI
Jul 7

Regime-Conditional Stabilisation of LLM-Augmented Cooperative Multi-Agent Reinforcement Learning

arXiv:2607. 04470v1 Announce Type: cross Abstract: Large Language Models (LLMs) offer a natural interface for translating human objectives into reward signals for cooperative multi-agent reinforcement learning (MARL), yet the training-time dynamics of this integration remain poorly understood.

By Faid Keddouri, Sohaib Houhou, Aissa Boulmerka, Nadir Farhi
llmsagentsreinforcement-learning
More like this →
arXiv Machine Learning
4d ago

EasyPPO: Stabilizing the Critic Is Key

arXiv:2609.36802v1 Announce Type: new Abstract: A key strength of Proximal Policy Optimization (PPO) is its learned critic, which uses historical trajectories collected during reinforcement learning...

By Xuanyi Zhou, Qiuyang Mang, Huanzhi Mao, Dacheng Li, Wenhao Chai, Mayank Mishra, Yichuan Wang, Karthik Narasimhan, Alvin Cheung, Joseph E. Gonzalez
llmsreinforcement-learning
More like this →