arXiv AI September 12, 2026 By Martin Klissarov, Akhil Bagaria, Ziyan Luo, George Konidaris, Doina Precup, Marlos C. Machado

Discovering Temporal Structure: An Overview of Hierarchical Reinforcement Learning

Read the original on arXiv AI →

The paper reviews hierarchical reinforcement learning (HRL) as a method for enabling agents to explore, plan, and learn in complex, open-ended environments by uncovering temporal structure in experience streams. It discusses the unclear definition of what makes a structure useful, the benefits of HRL for decision‑making challenges, and its impact on AI agent performance trade‑offs. The authors survey various HRL methods—from online learning to offline datasets and large language model integration—and outline the challenges and suitable domains for temporal structure discovery.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

arXiv AI

Jun 10

HIPIF: Hierarchical Planning and Information Folding for Long-Horizon LLM Agent Learning

arXiv:2606. 10507v1 Announce Type: new Abstract: While Large Language Models (LLMs) have demonstrated strong capabilities as autonomous agents across a wide range of tasks, their performance often degrades in multi-turn long-horizon agentic tasks.

By Juncheng Diao, Zhicong Lu, Peiguang Li, Yongwei Zhou, Changyuan Tian, Qingbin Li, Rongxiang Weng, Jingang Wang, Xunliang Cai

llms agents nlp reinforcement-learning benchmarks

arXiv AI

Aug 5

Hybrid LLM-Augmented Reinforcement Learning Agents for Complex Sequential Decision Tasks

arXiv:2608. 03502v1 Announce Type: new Abstract: Large Language Models (LLMs) have recently shown strong capabilities in reasoning, planning, and tool-use, enabling new forms of autonomous agents.

By Christophe D. Hounwanou, John Emeka Eze, Ya\'e Ulrich Gaba

llms agents reinforcement-learning

arXiv AI

Oct 9

Q-Shaped Options for Hierarchical Reinforcement Learning

The paper introduces Q-Shaped Options (QSO), a hierarchical reinforcement learning framework that learns action abstractions between consecutive levels using shared encoders shaped by Q functions. QSO employs distinct state-value functions, Q functions, and policies at each hierarchy level, encouraging the low-level Q function to preserve distinctions needed for optimal control while the high-level Q function discards unnecessary distinctions. Experiments on offline goal-conditioned locomotion and manipulation tasks show that QSO learns semantically meaningful option spaces and outperforms baselines, achieving non-zero performance where other algorithms fail.

By Clarisse Wibault, Antoine Gorceix, Antonio L\'eon Villares, Alexey Zakharov, Evangelos Chatzaroulas, Michael Matthews, Eduardo Pignatelli, Jakob Foerster

agents reinforcement-learning robotics

arXiv Machine Learning

Jul 31

TAPO: Transition-Aware Policy Optimization for LLM Agents

arXiv:2607. 27973v1 Announce Type: new Abstract: Recently, Reinforcement Learning (RL) has emerged as a crucial paradigm for the post-training of Large Language Model (LLM) agents.

By Cong Li, Peixi Peng, Yisen Zhao, Xinyu Hu, Shudong Liu, Zhan Su, Zhuojian Li

llms agents reinforcement-learning

arXiv Machine Learning

Jun 3

Multi$^2$: Hierarchical Multi-Agent Decision-Making with LLM-Based Agents in Interactive Environments

arXiv:2606. 03698v1 Announce Type: new Abstract: A central goal of large language model (LLM) research is to build agentic systems that can plan, act, and adapt through sustained interaction with dynamic environments.

By Sangeun Park, Minhae Kwon

llms agents reinforcement-learning fine-tuning benchmarks

arXiv Machine Learning

Jul 1

Hierarchical Message-Passing Policies for Multi-Agent Reinforcement Learning

arXiv:2507. 23604v2 Announce Type: replace Abstract: Decentralized Multi-Agent Reinforcement Learning (MARL) methods allow for learning scalable multi-agent policies, but suffer from partial observability and induced non-stationarity.

By Tommaso Marzi, Cesare Alippi, Andrea Cini

agents reinforcement-learning benchmarks

One email a morning, machine-written

Related stories

HIPIF: Hierarchical Planning and Information Folding for Long-Horizon LLM Agent Learning

Hybrid LLM-Augmented Reinforcement Learning Agents for Complex Sequential Decision Tasks

Q-Shaped Options for Hierarchical Reinforcement Learning

TAPO: Transition-Aware Policy Optimization for LLM Agents

Multi$^2$: Hierarchical Multi-Agent Decision-Making with LLM-Based Agents in Interactive Environments

Hierarchical Message-Passing Policies for Multi-Agent Reinforcement Learning