← Back to all news
arXiv AI By Guangzhi Wang, Junfu Pu, Yiran Wang, Xingyilang Yin, Honghao Chen, Cheng Cheng, Lingen Li, Yinan Zhou, Jiarui Wu, Yinuo Jiang, Hongbo Jin, Xiaodong Cun, Ying Shan

TaskFrame: Connecting Task-Level Planning with Frame-Level Control for Game Agents

Read the original on arXiv AI →

TaskFrame is a hierarchical planner-controller framework that links high-level task reasoning with low-level frame-by-frame keyboard-and-mouse control for game agents. It uses a vision-language task planner to break a goal into atomic instructions, generating continuous goal embeddings that the causal Transformer controller uses, along with recent observations, to predict discrete actions and continuous camera motion. Trained in three stages—planner training, controller pretraining, and joint alignment—TaskFrame demonstrates competitive instruction decomposition and higher combat success in Minecraft compared to baseline models.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

Related stories

Hugging Face Trending Papers
Aug 27

Decoupling Planning and Control for Instructable Agents

The paper introduces Instruct-to-Act, a system that decouples high‑level planning from low‑latency control by combining a vision‑language model (VLM) planner with a world‑model controller. The VLM generates sparse, high‑level text instructions, while the controller executes them autonomously at high frequency. Experiments across seven embodied environments, including multi‑agent settings, show that this approach outperforms both controller‑only and direct VLM action‑generation methods, maintains fast control, and allows swapping in different pretrained VLM planners without fine‑tuning.

llmsagentsreinforcement-learningroboticsfine-tuningmultimodal
More like this →
arXiv AI
Aug 28

Decoupling Planning and Control for Instructable Agents

The paper introduces Instruct-to-Act, a system that decouples planning and control by combining a vision‑language model (VLM) planner with a world‑model controller. The VLM generates sparse, high‑level text instructions, while the controller executes them at high frequency, trained via relabeling rollouts with synthetic instructions and joint optimization of behavior cloning, reward, and world‑model objectives. Across seven embodied environments—including multi‑agent settings—this approach outperforms controller‑only and direct VLM action methods, maintains fast control, and allows swapping pretrained VLM planners without fine‑tuning, achieving competitive results with strong baselines on most tasks.

By Zineng Tang, Kelsey R. Allen, Sjoerd van Steenkiste, Ishita Dasgupta, Alane Suhr
llmsagentsreinforcement-learningroboticsfine-tuningmultimodal
More like this →
arXiv Computer Vision
Oct 9

WorldGuide: Goal-Directed Video World Model for Procedural Task Execution

WorldGuide is a closed‑loop video world model that, given an initial image and a task goal, predicts an atomic action, generates the corresponding video clip, and uses that clip to decide the next action or terminate the task. It trains a Planner to predict the next action or completion from visual progress and an Executor to realize those actions, while a hierarchical visual memory tracks state over long horizons. The authors introduce WorldGuide‑Bench, a dataset of ~59K step‑annotated videos across 245 tasks, and show that WorldGuide outperforms the recent MiniMax‑H3 model on both WorldGuide‑Bench and VideoCraft‑Bench, highlighting the benefit of coupling planning with learned execution for goal‑directed procedural video generation.

By Ankan Deria, Komal Kumar, Hisham Cholakkal, Fahad Shahbaz Khan, Salman Khan
diffusion
More like this →