arXiv AI

Learning after Execution with Action-Specific Credit

arXiv Machine Learning
Oct 28

When Fine-Grained Credit Assignment Helps and Hurts: Multimodal Agentic Reinforcement Learning through Video Editing

The paper investigates how fine‑grained credit assignment affects learning in multimodal agentic reinforcement learning, using video editing as a testbed. Four movie‑trailer editing tasks—Content Selection, Narrative Sequencing, Rough Cut, and Fine Cut—are introduced to represent real‑world editing workflows with varying complexity and horizon lengths. The study finds that decaying process rewards significantly boost performance on the long‑horizon Fine Cut task, while offering modest gains on shorter tasks; however, fixed process‑reward weighting can lead to reward‑hacking and degraded outcomes, and on‑policy self‑distillation provides limited benefit.

By Lei Zhao, Zongheng Cao, Xinyu Hu
arXiv AI
Sep 1

Reconciling Process Supervision with Outcome-Based Credit in Agentic Policy Optimization

The paper introduces TASPO, a method that transforms privileged information (PI) into outcome‑grounded action credit for language‑model agents. TASPO constructs decision‑applicable PI from verified successful experience, aggregates PI‑induced likelihood shifts at the executable‑action level, and converts relative action support into positive, bounded, mean‑preserving weights on the original trajectory advantage. Experiments on three agentic benchmarks show TASPO improves over GRPO by 10.6% and generalizes better to unseen tasks, while reducing supervision mismatch and stabilizing policy optimization.

By Jingxiao Yang, Wangjie Gan, Yingxuan Zhuang, Wenqi Zhang, Jintao Chen, Xuhong Zhang
arXiv AI
Oct 2

Measuring the Stability Assumption Behind Action Chunking

The paper investigates how small action errors evolve when using action chunking in behavioural cloning. By injecting errors at each state and observing their growth under open‑loop (no replanning) and closed‑loop (replanning) regimes, the authors classify states as contracting, expanding, or unresolved. Across twelve manipulation tasks, they find that stable states are rare, error amplification is common, and that short‑horizon fitting can overestimate long‑horizon propagation. Predictors trained on camera and proprioceptive data can recover open‑loop stability but only partially capture closed‑loop dynamics, indicating that standard imitation learning does not reliably produce policies that contract errors when perturbed.

By Aryan Goyal
arXiv AI
Oct 1

Trust the Critic More

The paper introduces Actor‑Critic with Action Chunking (AC2), a method that assigns credit to short action chunks instead of entire trajectories, enabling policy updates without waiting for terminal rewards. AC2 employs local readiness, reference solutions, and 10k‑token chunks to make critic‑based credit assignment reliable. Experiments on Qwen3‑4B with FineProofs‑RL show AC2 surpasses GRPO’s peak validation score while using 2.5× fewer decoding FLOPs and fewer training steps.

By Kaiyue Wen, Luke Bailey, Arvind Mahankali, Tengyu Ma
arXiv Machine Learning
Aug 31

VICT: Verifier-Instrumented Credit Tracing for Long-Horizon LLM Agent Reinforcement Learning

The paper introduces VICT, a method that leverages the internal structure of verifiable tasks to perform fine‑grained credit assignment for long‑horizon LLM agents. VICT exposes executable or evidence‑backed atoms from a task’s terminal verifier and traces them back to actions via dependency‑valid proof edges, redistributing advantage only along these edges. This approach improves performance on ALFWorld and WebShop compared to outcome‑only training and matches recent fine‑grained credit methods without requiring additional critics, labels, or inference‑time verifier access.

By Pengcheng Li, Zhengyang Zhang, Dongxu Zhang, Sui Huang, Shaohua Ma
arXiv Computer Vision
Oct 9

WorldGuide: Goal-Directed Video World Model for Procedural Task Execution

WorldGuide is a closed‑loop video world model that, given an initial image and a task goal, predicts an atomic action, generates the corresponding video clip, and uses that clip to decide the next action or terminate the task. It trains a Planner to predict the next action or completion from visual progress and an Executor to realize those actions, while a hierarchical visual memory tracks state over long horizons. The authors introduce WorldGuide‑Bench, a dataset of ~59K step‑annotated videos across 245 tasks, and show that WorldGuide outperforms the recent MiniMax‑H3 model on both WorldGuide‑Bench and VideoCraft‑Bench, highlighting the benefit of coupling planning with learned execution for goal‑directed procedural video generation.

By Ankan Deria, Komal Kumar, Hisham Cholakkal, Fahad Shahbaz Khan, Salman Khan
arXiv Machine Learning
Jul 21

Foresight Residual RL for Long-Horizon Robot Manipulation with Vision-Language-Action Models

arXiv:2607. 16506v1 Announce Type: cross Abstract: Vision-Language-Action (VLA) policies offer strong general-purpose manipulation priors, but often fail on tight-tolerance, contact-rich assembly due to long-horizon credit assignment and subtask coupling: a state that is geometrically successful for the current skill can be brittle for downstream skills.

By Yuhan Liu, Xinyu Zhang, Litao Liu, Abdeslam Boularias
arXiv Machine Learning
Sep 22

Success Leaves Detours: Learning Executable Walkthroughs for Long-Horizon Agents

The paper introduces Trace, a framework that transforms sparse-reward trajectories into executable walkthroughs by identifying progress anchors, propagating credit, and estimating action prerequisites. Trace compiles noisy trajectories into state‑conditioned, verifiable procedures that remove loops and detours, enabling reuse, intermediate‑state resumption, and programmatic verification. Experiments on J‑TTL, WebShop, and ScienceWorld with three open‑source LLMs show that Trace outperforms eight baselines, improving average AUC and Final‑$3$ by 30.0% and 40.5% while using fewer inference tokens.

By Kaijie Chen, Chenyu Fang, Liang Yan, Bo Li, Bo Zhang, Peng Ye