The paper investigates whether a frozen recurrent network can provide sufficient memory for partially observable Markov decision processes (POMDPs). Using a fixed recurrent network combined with a trainable output module, the authors demonstrate that such networks can recall information over hundreds of steps and achieve competitive control performance across diverse POMDP tasks, while using fewer than half the trainable parameters of fully trainable recurrent networks. These findings suggest that learning the recurrent memory module is not always necessary, offering a simple, parameter‑efficient baseline for partially observable reinforcement learning.
By Zekang Wang, Zhe He, Gaojie Jin
arXiv:2609.38598v1 Announce Type: new
Abstract: Partially observable environments pose a fundamental challenge in deep reinforcement learning, requiring agents to compress temporal information from o...
By Sathya Kamesh Bhethanabhotla, Efstratios Gavves, Andr\'e Biedenkapp
arXiv:2605. 24709v2 Announce Type: replace Abstract: Streaming reinforcement learning has emerged as an online learning paradigm that conforms to the restrictions of natural learning agents that process data incrementally, i.
By Noah Farr, Aryaman Reddi, Carlo D'Eramo, Jan Peters
The paper introduces memoroids, a monoid‑based framework that reformulates memory models like RNNs and Transformers for better handling of long sequences in reinforcement learning. It critiques traditional batching approaches in recurrent RL, identifies their shortcomings, and proposes a new batching method using memoroids that enhances sample efficiency, boosts return, and simplifies recurrent loss implementation.
By Steven Morad, Chris Lu, Ryan Kortvelesy, Stephan Liwicki, Jakob Foerster, Amanda Prorok
The paper introduces ELiSe, a model that leverages cortical network scaffolds and dendritic compartments to learn complex non‑Markovian spatio‑temporal patterns using only local, always‑on, phase‑free synaptic plasticity. It demonstrates the model’s ability to acquire and replay intricate sequences, exemplified by a birdsong learning mock‑up, and shows robustness to external disturbances and flexibility in parameter settings.
By Laura Kriener, Kristin V\"olk, Ben von H\"unerbein, Federico Benitez, Walter Senn, Mihai A. Petrovici
arXiv:2603. 23461v2 Announce Type: replace Abstract: We study reinforcement learning (RL) with linear function approximation in Markov Decision Processes (MDPs) satisfying \emph{linear Bellman completeness} -- a fundamental setting where the Bellman backup of any linear value function remains linear.
By Zakaria Mhammedi, Alexander Rakhlin, Nneka Okolo
arXiv:2605.07057v2 Announce Type: replace
Abstract: Online reinforcement learning (RL) relies on the Markov property for guaranteed performance, but real-world applications often lack well-defined st...
By Jiamin Xu, Jacqueline Maasch, Kyra Gan
The paper investigates reinforcement learning in constrained partially observable Markov decision processes (CPOMDPs), where an agent must maximize reward while respecting a cost constraint using incomplete state information. By approximating the CPOMDP with finite-memory models and treating observation-action windows as states of a constrained MDP, the authors develop a primal–dual algorithm that offers the first non-asymptotic guarantees on both average reward suboptimality and constraint violation. Under a filter‑stability assumption, they show that the approximation errors decay geometrically with the window length, and validate their method with experiments in partially observable gridworlds.
By Valentina Zaccaria, Philip Jordan, Tingting Ni, Gian Antonio Susto, Maryam Kamgarpour
arXiv:2602. 01196v2 Announce Type: replace Abstract: Recurrent neural policies are widely used in partially observable control and meta-RL tasks.
By Jin Li, Yue Wu, Mengsha Huang, Yuhao Sun, Hao He, Xianyuan Zhan
The paper introduces Keyframe Mnemonics, a self‑supervised approach that identifies critical observations (mnemonics) to aid behavior cloning in non‑Markovian settings. By training a policy conditioned on these keyframes, the method achieves perfect success rates on synthetic memory tasks and improves performance on a real‑robot manipulation benchmark, maintaining high success even at horizons far beyond training. The approach offers theoretical context‑retention guarantees and practical scalability to long‑horizon decision making.
By Prabin Kumar Rath, Omkar Patil, Nakul Gopalan
The paper introduces a hierarchical model-based reinforcement learning framework that learns control policies from action-free time series by exploiting shared dynamics across related systems. It uses low-dimensional dynamical embeddings to capture both shared structure and individual variation, which then parameterize shared policy and value networks. Experiments on Lorenz-63, double-pendulum, and neural-behavioral data show that these hierarchical policies outperform independently trained ones, achieve higher rewards than planning with the same models, and generalize to unseen systems using only embedding inference.
By Niklas Emonds, Georgia Koppe
arXiv:2601. 18930v4 Announce Type: replace-cross Abstract: We are interested in enabling autonomous agents to learn and reason about systems with hidden states, such as locking mechanisms.
By Seiji Shaw, Travis Manderson, Chad Kessens, Nicholas Roy