Q($\gamma_H$) introduces a multi-horizon Q-learning framework that keeps separate discount-specific heads and blends them during control using an entropy-derived confidence signal. The adaptive mixing shifts from short to long horizons as policy confidence grows, allowing the agent to adjust its planning horizon without altering individual Bellman updates. Experiments on classical control and deep reinforcement‑learning tasks show the method excels on delayed‑credit problems compared to fixed‑discount baselines.
By Arwa Albelaihi, Abdulrahman Altahhan
arXiv:2607. 20656v1 Announce Type: cross Abstract: Effective decision-making in complex and changing environments requires balancing short-term and long-term consequences.
By Manoosh Samiei, Doina Precup, Paul Masset
arXiv:2605.06149v2 Announce Type: replace-cross
Abstract: The discount factor in reinforcement learning controls both the effective planning horizon and the strength of bootstrapping, yet most deep R...
By Yaomin Wang, Jianting Pan, Ran Tian, Xiaoyang Li, Yu Zhang, Hengle Qin, Tianshu YU
arXiv:2607. 28916v1 Announce Type: cross Abstract: Multistep credit assignment is critical for sample-efficient reinforcement learning, yet managing off-policy bias in Q-learning remains a fundamental challenge.
By Brett Daley
The paper identifies an informational curse of horizon in goal-conditioned policy learning, showing that longer goal relabeling horizons during training can hurt policy generalization and performance. Controlled experiments with oracle planners separate training horizons from test horizons, revealing that behavioral cloning policies degrade with longer horizons while reinforcement learning mitigates this effect. The authors attribute this to reduced conditional mutual information between actions and hindsight-relabeled goals, and demonstrate that distilling short-horizon policy Jacobians into long-horizon policies improves performance, especially in combinatorial manipulation tasks.
By John L. Zhou, Yuxuan Dong, Jonathan C. Kao
In reinforcement learning, advantage-based value-learning methods, unlike standard Q-learning, share information across actions by decomposing the action-value function into a state-dependent value co...
arXiv:2607. 23726v1 Announce Type: cross Abstract: Exploration in sparse-reward long-horizon tasks poses significant challenges for reinforcement learning.
By Zahra Abdalla Elashaal, Afef Hfaiedh, Nahla Khraief, Issmail Ellabib, Giansalvo Cirrincione
arXiv:2608. 08158v1 Announce Type: new Abstract: Sparse, delayed, and weakly informative rewards remain central obstacles to efficient reinforcement learning.
By Fouad Bahrpeyma
arXiv:2610.14988v1 Announce Type: cross
Abstract: The average-reward setting for reinforcement learning is challenging because its Bellman operator is known to contract in span only when the underlyi...
By Julien Roger, Sam Bodmer, Jian Qian, Benjamin Riviere
V‑SHARE is an advantage‑based value‑learning algorithm that blends cross‑action value sharing with an importance‑weighted Bellman‑residual correction. The paper presents the first full theoretical analysis of such an algorithm, offering both asymptotic convergence and finite‑sample guarantees. It also introduces schedules that balance deterministic contraction against importance‑weighting noise, and a two‑timescale analysis that eliminates a residual‑weight threshold while maintaining convergence.
By Kintan Saha, Hamed Masoudi, Xinyi Sheng, Dominik Baumann, Joni Pajarinen, Arsenii Mustafin
Offline goal-conditioned reinforcement learning (RL) holds the promise of learning general-purpose policies from static datasets. However, scaling these methods to long-horizon tasks remains a challenge due to the curse of horizon, where value estimation errors can compound through long chains of bootstrapped Bellman backups.
arXiv:2606. 05885v1 Announce Type: new Abstract: Long-horizon LLM agents require reinforcement learning methods that can assign credit to intermediate decisions under sparse and delayed rewards.
By Yuanfan Li, Qi Zhou, Wenjing Duan, Lu Chen