arXiv AI September 28, 2026 By Taha Shieenavaz, Shabnam Zareshahraki, Loris Nanni

A Progressive Design Study of Visual Encoders and Value Estimation for Replay-Free Parallelized Q-Learning

The paper presents a three‑phase study of visual encoders and value‑estimation methods for replay‑free parallelized Q‑learning within the Parallelized Q‑Network framework. It compares eight convolutional encoder topologies, adds Hadamax‑style multiplicative interactions and pooling, and evaluates categorical‑dueling, ensemble‑dueling, and categorical ensemble‑dueling value‑estimation configurations. The resulting architecture, Aftab, outperforms a baseline PQN on Atari‑57 and shows improved performance on Procgen Hard, demonstrating that visual topology, multiplicative representation, and downstream value‑estimation design significantly influence replay‑free Q‑learning when considered alongside computational complexity.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

reinforcement-learning

arXiv AI

Sep 25

Aftab: A Progressive Design Study of Visual Encoders and Value Estimation for Replay-Free Parallelized Q-Learning

The paper presents Aftab, a new architecture for replay‑free parallelized Q‑learning that systematically explores visual encoder designs, multiplicative feature interactions, and value‑estimation strategies. Through a three‑phase study on Atari‑57, the authors compare eight convolutional encoders, integrate Hadamax‑style interactions, and evaluate categorical‑dueling, ensemble‑dueling, and combined configurations, ultimately achieving a higher human‑normalized score than the baseline PQN. Aftab is also evaluated on Procgen Hard, showing improved terminal IQM and learning‑curve area, and the full framework is released as open source.

By Taha Shieenavaz, Shabnam Zareshahraki, Loris Nanni

reinforcement-learning

arXiv AI

Aug 10

Aftab: A Comprehensive Benchmark of CNN Encoders and Advanced Value Functions in Parallelized Q-Networks

arXiv:2608. 07335v1 Announce Type: cross Abstract: Recent advancements in deep reinforcement learning have increasingly favored simplified, highly parallelized paradigms.

By Taha Shieenavaz, Shabnam Zareshahraki, Loris Nanni

reinforcement-learning benchmarks

arXiv Machine Learning

Jul 23

Memory Merge DQN: Sensitivity Weighted Target Updates for Stable Value Learning

arXiv:2607. 19397v1 Announce Type: new Abstract: Deep Q-networks use target networks to stabilise bootstrapped value learning, but the standard hard copy update also introduces a tradeoff.

By Adrian Ly, Richard Dazeley, Peter Vamplew, Sunil Aryal, Francisco Cruz

agents

arXiv Machine Learning

Sep 22

Streaming Deep Reinforcement Learning Finally Works

arXiv:2410.14606v3 Announce Type: replace Abstract: Learning from a stream of experience as it arrives, also known as streaming learning, is a core part of natural learning. However, reliable streami...

By Mohamed Elsayed, Elena Sorina Lupu, Gautham Vasan, A. Rupam Mahmood

reinforcement-learning robotics benchmarks

arXiv Machine Learning

Aug 4

Upper-Expectile Multi-Step Q-Learning for Off-Policy Reinforcement Learning

arXiv:2608. 02034v1 Announce Type: new Abstract: Multi-step returns accelerate reward propagation in off-policy reinforcement learning, but couple the evaluation of each decision to the suboptimal logged actions that follow it, inducing a pessimistic bias that grows with the horizon.

By Abdelghani Ghanem, Mounir Ghogho

reinforcement-learning robotics efficiency safety

arXiv AI

Sep 10

On BatchNorm Forward Modes in Value-Based Reinforcement Learning

arXiv:2609.06421v1 Announce Type: cross Abstract: Batch normalization (BN) substantially improves sample efficiency in continuous-control actor-critic methods such as CrossQ, yet recent studies repor...

By Daniel Palenicek, Mikael Henaff, Scott Fujimoto, Koustuv Sinha

reinforcement-learning

One email a morning, machine-written

Related stories

Aftab: A Progressive Design Study of Visual Encoders and Value Estimation for Replay-Free Parallelized Q-Learning

Aftab: A Comprehensive Benchmark of CNN Encoders and Advanced Value Functions in Parallelized Q-Networks

Memory Merge DQN: Sensitivity Weighted Target Updates for Stable Value Learning

Streaming Deep Reinforcement Learning Finally Works

Upper-Expectile Multi-Step Q-Learning for Off-Policy Reinforcement Learning

On BatchNorm Forward Modes in Value-Based Reinforcement Learning