arXiv Machine Learning October 7, 2026 By Chung Min Kim, Brent Yi, David McAllister, Hongsuk Choi, Himanshu Gaurav Singh, Jinkun Cao, Ken Goldberg, Pieter Abbeel, Carmelo Sferrazza, Angjoo Kanazawa

QF3: Fast Flow RL with Filtered Q-Gradients

Read the original on arXiv Machine Learning →

QF3 (Fast Flow RL with Filtered Q-Gradients) is an online off‑policy reinforcement learning algorithm that trains flow policies by combining flow matching with the critic’s action gradient, backpropagated through a one‑step prediction of the flow output. It selectively applies the critic gradient only to action dimensions close to the replay action to ensure reliable updates. QF3 is the first off‑policy flow RL method to train humanoid locomotion policies from scratch and transfer them zero‑shot to hardware, achieving a 10× wall‑clock speedup over the on‑policy FPO++ method, and it can also fine‑tune pretrained flow‑based manipulation policies on ABC‑Sim and Robomimic tasks.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

Hugging Face Trending Papers

6d ago

QF3: Fast Flow RL with Filtered Q-Gradients

Flow policies have become a standard policy class for learning robot behaviors from demonstrations, but reinforcement learning is still critical for improving pre-trained flow policies or learning the...

diffusion reinforcement-learning robotics fine-tuning efficiency

arXiv Machine Learning

5d ago

Q-Learning with Scalar Adjoint Matching

The paper introduces Q‑Learning with Scalar Adjoint Matching (SQAM), a method that replaces expensive per‑step vector–Jacobian products in flow policy fine‑tuning with a closed‑form scalar adjoint that scales the value gradient by flow time. SQAM also incorporates a value penalty on policy‑generated actions, which is shown to be crucial for effective learning. Experiments demonstrate that SQAM outperforms strong baselines on the hardest OGBench domains and improves supervised fine‑tuning on a real bimanual robot with a vision‑language‑action policy.

By Yonghoon Dong, Minsung Yoon, Jaehyuk Kim, Jungwoo Park, Changyeon Kim, Jinwoo Shin

reinforcement-learning robotics fine-tuning multimodal

arXiv AI

Jun 10

Test-Time Gradient Guidance of Flow Policies in Reinforcement Learning

arXiv:2606. 11087v1 Announce Type: cross Abstract: Expressive continuous control policies, such as diffusion and flow models, form the backbone of recent advances in scaling imitation learning for simulated and real robot control.

By Zhiyuan Zhou, Andy Peng, Charles Xu, Qiyang Li, Tobias Springenberg, Kevin Frans, Sergey Levine

diffusion reinforcement-learning robotics benchmarks

arXiv Machine Learning

Jul 31

REFINE-DP: Diffusion Policy Fine-tuning for Humanoid Loco-manipulation via Reinforcement Learning

arXiv:2603. 13707v3 Announce Type: replace-cross Abstract: Humanoid loco-manipulation requires coordinated task-space motion planning with stable loco-manipulation command tracking under complex robot-environment dynamics and long-horizon tasks.

By Zhaoyuan Gu, Yipu Chen, Zimeng Chai, Alfred Cueva, Thong Nguyen, Yifan Wu, Huishu Xue, Minji Kim, Isaac Legene, Fukang Liu, KyoungMok Kim, Ayan Barula, Yongxin Chen, Ye Zhao

diffusion reinforcement-learning robotics fine-tuning

arXiv AI

Aug 18

Max-Q Selective Imitation for Human-in-the-Loop Online Robot Learning

arXiv:2608. 15088v1 Announce Type: cross Abstract: Human-in-the-loop (HIL) online reinforcement learning for real robots must absorb human interventions quickly while continuing to improve beyond the human prior.

By Zihang Wang, Yishan Wang

agents reinforcement-learning robotics

arXiv AI

Jun 9

Reinforcement Learning for Flow-Matching Policies with Density Transport

arXiv:2606. 08602v1 Announce Type: cross Abstract: We present an online reinforcement learning (RL) algorithm for fine-tuning flow-matching policies in continuous-control problems.

By Boshu Lei, Kostas Daniilidis, Antonio Loquercio

diffusion reinforcement-learning robotics fine-tuning efficiency multimodal safety

One email a morning, machine-written

Related stories

QF3: Fast Flow RL with Filtered Q-Gradients

Q-Learning with Scalar Adjoint Matching

Test-Time Gradient Guidance of Flow Policies in Reinforcement Learning

REFINE-DP: Diffusion Policy Fine-tuning for Humanoid Loco-manipulation via Reinforcement Learning

Max-Q Selective Imitation for Human-in-the-Loop Online Robot Learning

Reinforcement Learning for Flow-Matching Policies with Density Transport