← Back to all news
arXiv AI By Yuan Zhuang, Yuexin Bian, Sihong He, Jie Feng, Qing Su, Songyang Han, Jonathan Petit, Shihao Ji, Yuanyuan Shi, Fei Miao

Low-Rank Adaptation for Critic Learning in Off-Policy Reinforcement Learning

Read the original on arXiv AI →

The paper proposes using Low‑Rank Adaptation (LoRA) as a structural regularizer for critic learning in off‑policy reinforcement learning. By freezing randomly initialized base matrices and optimizing only low‑rank adapters, the method constrains weight changes to low‑rank residuals, reducing critic loss and improving policy performance. Experiments on state‑based continuous‑control tasks with SAC, FastTD3, and their SimbaV2 variants show that LoRA achieves the best or competitive results on most tasks.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

Related stories

arXiv AI
Sep 17

Rethinking Critic Learning in PPO: Understanding and Mitigating Value Flattening

The paper identifies a failure mode in Proximal Policy Optimization (PPO) critics called Value Flattening, where state values vary sharply across intermediate states while critic predictions remain flat. The authors analyze this phenomenon theoretically and empirically, linking it to an implicit variance penalty and redundant updates from temporally correlated states. They propose Sparse Proximal Policy Optimization (SP³O), which applies the value loss to only a few well‑separated states, and demonstrate that this sparse supervision mitigates Value Flattening and improves policy performance on Qwen3‑Base across model sizes and evaluation suites.

By Yizhuo Li, Jianhao Yan, Yun Luo, Zhi Wang, Futing Wang, Rong-Xi Tan, Kanghui Tian, Ganqu Cui, Ning Ding, Peilin Zhao, Yafu Li, Yu Cheng
llmsreinforcement-learning
More like this →