Q($\gamma_H$) introduces a multi-horizon Q-learning framework that keeps separate discount-specific heads and blends them during control using an entropy-derived confidence signal. The adaptive mixing shifts from short to long horizons as policy confidence grows, allowing the agent to adjust its planning horizon without altering individual Bellman updates. Experiments on classical control and deep reinforcement‑learning tasks show the method excels on delayed‑credit problems compared to fixed‑discount baselines.
A fixed discount factor in Q-learning imposes a single planning horizon throughout training, despite the differing demands of early exploration and long-term credit assignment. We propose Q($γ_H$), a...
arXiv:2607. 20656v1 Announce Type: cross Abstract: Effective decision-making in complex and changing environments requires balancing short-term and long-term consequences.
arXiv:2605.06149v2 Announce Type: replace-cross Abstract: The discount factor in reinforcement learning controls both the effective planning horizon and the strength of bootstrapping, yet most deep R...
arXiv:2607. 28916v1 Announce Type: cross Abstract: Multistep credit assignment is critical for sample-efficient reinforcement learning, yet managing off-policy bias in Q-learning remains a fundamental challenge.
The paper identifies an informational curse of horizon in goal-conditioned policy learning, showing that longer goal relabeling horizons during training can hurt policy generalization and performance. Controlled experiments with oracle planners separate training horizons from test horizons, revealing that behavioral cloning policies degrade with longer horizons while reinforcement learning mitigates this effect. The authors attribute this to reduced conditional mutual information between actions and hindsight-relabeled goals, and demonstrate that distilling short-horizon policy Jacobians into long-horizon policies improves performance, especially in combinatorial manipulation tasks.
In reinforcement learning, advantage-based value-learning methods, unlike standard Q-learning, share information across actions by decomposing the action-value function into a state-dependent value co...
arXiv:2610.14988v1 Announce Type: cross Abstract: The average-reward setting for reinforcement learning is challenging because its Bellman operator is known to contract in span only when the underlyi...
arXiv:2608. 08158v1 Announce Type: new Abstract: Sparse, delayed, and weakly informative rewards remain central obstacles to efficient reinforcement learning.
V‑SHARE is an advantage‑based value‑learning algorithm that blends cross‑action value sharing with an importance‑weighted Bellman‑residual correction. The paper presents the first full theoretical analysis of such an algorithm, offering both asymptotic convergence and finite‑sample guarantees. It also introduces schedules that balance deterministic contraction against importance‑weighting noise, and a two‑timescale analysis that eliminates a residual‑weight threshold while maintaining convergence.
arXiv:2607. 23726v1 Announce Type: cross Abstract: Exploration in sparse-reward long-horizon tasks poses significant challenges for reinforcement learning.
arXiv:2511. 03836v2 Announce Type: replace Abstract: Deep Q-Networks (DQNs) estimate future returns by learning from transitions sampled from a replay buffer.
Offline goal-conditioned reinforcement learning (RL) holds the promise of learning general-purpose policies from static datasets. However, scaling these methods to long-horizon tasks remains a challenge due to the curse of horizon, where value estimation errors can compound through long chains of bootstrapped Bellman backups.