arXiv AI By Matteo Merler, Giovanni Bonetta, Davide Zago, Rossella Cancelliere, Bernardo Magnini

Selective Agent Guidance via Entropy: Learning Autonomous Policies from Imperfect VLM Teachers

Read the original on arXiv AI →

The paper introduces SAGE, a framework that selectively queries a Vision‑Language Model (VLM) teacher only when the learner is uncertain, using the teacher’s suggestions to guide training and distill them into a lightweight reinforcement learning policy. SAGE weights teacher actions by environment‑derived advantages, allowing the policy to improve beyond the imperfect VLM. Experiments on sparse‑reward visual reasoning and navigation tasks show that the learned policies can act without VLM guidance at evaluation, reduce VLM usage during training, and sometimes outperform the teacher itself.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

arXiv AI
2d ago

Prompt-Driven Exploration: Language as an Exploration Space for VLA Reinforcement Learning

arXiv:2607.08837v4 Announce Type: replace-cross Abstract: Exploration is essential to RL since a policy cannot improve by repeatedly sampling the behaviors it already prefers. Standard methods inject...

By Sunshine Jiang, John Marangola, David Zhang, Raghuram Kowdeed, Ruiyang Luo, Nitish Dashora, Richard Li, Pulkit Agrawal, Zhang-Wei Hong
arXiv AI
Aug 28

Decoupling Planning and Control for Instructable Agents

The paper introduces Instruct-to-Act, a system that decouples planning and control by combining a vision‑language model (VLM) planner with a world‑model controller. The VLM generates sparse, high‑level text instructions, while the controller executes them at high frequency, trained via relabeling rollouts with synthetic instructions and joint optimization of behavior cloning, reward, and world‑model objectives. Across seven embodied environments—including multi‑agent settings—this approach outperforms controller‑only and direct VLM action methods, maintains fast control, and allows swapping pretrained VLM planners without fine‑tuning, achieving competitive results with strong baselines on most tasks.

By Zineng Tang, Kelsey R. Allen, Sjoerd van Steenkiste, Ishita Dasgupta, Alane Suhr
Hugging Face Trending Papers
Aug 27

Decoupling Planning and Control for Instructable Agents

The paper introduces Instruct-to-Act, a system that decouples high‑level planning from low‑latency control by combining a vision‑language model (VLM) planner with a world‑model controller. The VLM generates sparse, high‑level text instructions, while the controller executes them autonomously at high frequency. Experiments across seven embodied environments, including multi‑agent settings, show that this approach outperforms both controller‑only and direct VLM action‑generation methods, maintains fast control, and allows swapping in different pretrained VLM planners without fine‑tuning.

arXiv Machine Learning
Jul 22

H$^2$SD: Hybrid Hindsight Self-Distillation

arXiv:2607. 18955v1 Announce Type: new Abstract: Reinforcement learning with verifiable rewards (RLVR) has substantially improved the reasoning capabilities of large language models on tasks such as mathematical reasoning and code generation.

By Qiye Cai, Yichuan Ma, Linyang Li, Peiji Li, Yongkang Chen, Qipeng Guo, Yicheng Zou, Tao Gui, Xiaocheng Feng, Bing Qin