arXiv Machine Learning October 8, 2026 By Yonghoon Dong, Minsung Yoon, Jaehyuk Kim, Jungwoo Park, Changyeon Kim, Jinwoo Shin

Q-Learning with Scalar Adjoint Matching

Read the original on arXiv Machine Learning →

The paper introduces Q‑Learning with Scalar Adjoint Matching (SQAM), a method that replaces expensive per‑step vector–Jacobian products in flow policy fine‑tuning with a closed‑form scalar adjoint that scales the value gradient by flow time. SQAM also incorporates a value penalty on policy‑generated actions, which is shown to be crucial for effective learning. Experiments demonstrate that SQAM outperforms strong baselines on the hardest OGBench domains and improves supervised fine‑tuning on a real bimanual robot with a vision‑language‑action policy.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

arXiv Machine Learning

6d ago

QF3: Fast Flow RL with Filtered Q-Gradients

QF3 (Fast Flow RL with Filtered Q-Gradients) is an online off‑policy reinforcement learning algorithm that trains flow policies by combining flow matching with the critic’s action gradient, backpropagated through a one‑step prediction of the flow output. It selectively applies the critic gradient only to action dimensions close to the replay action to ensure reliable updates. QF3 is the first off‑policy flow RL method to train humanoid locomotion policies from scratch and transfer them zero‑shot to hardware, achieving a 10× wall‑clock speedup over the on‑policy FPO++ method, and it can also fine‑tune pretrained flow‑based manipulation policies on ABC‑Sim and Robomimic tasks.

By Chung Min Kim, Brent Yi, David McAllister, Hongsuk Choi, Himanshu Gaurav Singh, Jinkun Cao, Ken Goldberg, Pieter Abbeel, Carmelo Sferrazza, Angjoo Kanazawa

diffusion reinforcement-learning robotics fine-tuning efficiency

arXiv AI

Jun 10

Test-Time Gradient Guidance of Flow Policies in Reinforcement Learning

arXiv:2606. 11087v1 Announce Type: cross Abstract: Expressive continuous control policies, such as diffusion and flow models, form the backbone of recent advances in scaling imitation learning for simulated and real robot control.

By Zhiyuan Zhou, Andy Peng, Charles Xu, Qiyang Li, Tobias Springenberg, Kevin Frans, Sergey Levine

diffusion reinforcement-learning robotics benchmarks

Hugging Face Trending Papers

6d ago

QF3: Fast Flow RL with Filtered Q-Gradients

Flow policies have become a standard policy class for learning robot behaviors from demonstrations, but reinforcement learning is still critical for improving pre-trained flow policies or learning the...

diffusion reinforcement-learning robotics fine-tuning efficiency

arXiv AI

Jun 9

Reinforcement Learning for Flow-Matching Policies with Density Transport

arXiv:2606. 08602v1 Announce Type: cross Abstract: We present an online reinforcement learning (RL) algorithm for fine-tuning flow-matching policies in continuous-control problems.

By Boshu Lei, Kostas Daniilidis, Antonio Loquercio

diffusion reinforcement-learning robotics fine-tuning efficiency multimodal safety

arXiv AI

Jun 16

QPILOTS: Efficient Test-Time Q-Steering for Flow Policies

arXiv:2606. 14801v1 Announce Type: cross Abstract: Flow-matching and diffusion policies are expressive action generators, but optimizing them with temporal-difference reinforcement learning (RL) remains difficult.

By Yifan Ruan, Chenyang Cao, Andreas Burger, Ali Pesaranghader, Kaveh Kamali, Jaehong Kim, Nandita Vijaykumar, Alan Aspuru-Guzik, Igor Gilitschenski, Nicholas Rhinehart

diffusion reinforcement-learning robotics fine-tuning multimodal benchmarks

arXiv Machine Learning

Jun 26

Reinforcement Fine-Tuning of Flow-Matching Policies for Vision-Language-Action Models

arXiv:2510. 09976v2 Announce Type: replace Abstract: Vision-Language-Action (VLA) models such as OpenVLA, Octo, and $\pi_0$ have shown strong generalization by leveraging large-scale demonstrations, yet their performance is still fundamentally constrained by the quality and coverage of supervised data.

By Mingyang Lyu, Yinqian Sun, Erliang Lin, Huangrui Li, Ruolin Chen, Feifei Zhao, Yi Zeng

diffusion reinforcement-learning fine-tuning multimodal benchmarks

One email a morning, machine-written

Related stories

QF3: Fast Flow RL with Filtered Q-Gradients

Test-Time Gradient Guidance of Flow Policies in Reinforcement Learning

QF3: Fast Flow RL with Filtered Q-Gradients

Reinforcement Learning for Flow-Matching Policies with Density Transport

QPILOTS: Efficient Test-Time Q-Steering for Flow Policies

Reinforcement Fine-Tuning of Flow-Matching Policies for Vision-Language-Action Models