arXiv Machine Learning
Jul 3
Learning the Supports for Categorical Critic in Reinforcement Learning
arXiv:2607. 01880v1 Announce Type: new Abstract: Value functions are an essential component in actor-critic based deep reinforcement learning (RL).
Learning the Supports for Categorical Critic in Reinforcement Learning
Value functions are an essential component in actor-critic based deep reinforcement learning (RL). Conventionally, these functions are trained as a regression task by minimising the mean squared error (MSE) relative to bootstrapped target values.
Related stories
arXiv Machine Learning
Jul 30
Collaborative Weighting with Pessimistic Critic for Mitigating Overestimation in Off-Policy Reinforcement Learning
arXiv:2607. 26509v1 Announce Type: new Abstract: Deep off-policy reinforcement learning algorithms for continuous control typically rely on neural value function approximation to guide policy improvement.
arXiv Machine Learning
Jun 25
RN-D: Discretized Categorical Actors for On-Policy Reinforcement Learning
arXiv:2601. 23075v2 Announce Type: replace Abstract: On-policy Reinforcement Learning (RL) remains a dominant paradigm for continuous control, yet standard implementations rely on Gaussian actors and relatively shallow MLP policies, often leading to brittle optimization when gradients are noisy, and policy updates must be conservative.
arXiv Machine Learning
Aug 4
Start Classifying: Categorical Critics for LLM Reinforcement Learning
arXiv:2608. 02181v1 Announce Type: new Abstract: Proximal Policy Optimization (PPO) for large language models typically trains its critic by mean-squared-error (MSE) regression on scalar value targets.
arXiv Machine Learning
Aug 24
Decoupling Policy Extraction for Offline Reinforcement Learning
arXiv:2608.20909v1 Announce Type: new Abstract: Offline RL methods commonly jointly train the actor and critic, where the critic is used to guide the actor toward higher-value actions. This coupled l...
Hugging Face Trending Papers
Sep 23
Limiting-Kernel Q($λ$): Bridging Short and Long Horizons
In value-based reinforcement learning, improving the accuracy of policy evaluation has been shown to improve downstream policy optimization performance. The widely adopted family of approximations rel...
arXiv Machine Learning
Aug 4
Analytic Planning under Uncertainty with Moment Closure
arXiv:2608. 02519v1 Announce Type: new Abstract: Effective model-based reinforcement learning in stochastic environments requires planning that accounts for predictive uncertainty.
Hugging Face Trending Papers
Aug 3
Analytic Planning under Uncertainty with Moment Closure
Effective model-based reinforcement learning in stochastic environments requires planning that accounts for predictive uncertainty. Propagating full state distributions analytically offers a principled way to do this, but has traditionally required restrictive policy or reward structures to remain tractable.
arXiv Machine Learning
Jul 27
Generalized Gaussian Temporal Difference Error for Uncertainty-aware Reinforcement Learning
arXiv:2408. 02295v4 Announce Type: replace Abstract: Conventional uncertainty-aware temporal difference (TD) learning often models TD errors as zero-mean Gaussian.
arXiv Machine Learning
Jul 22
Gaussian-Mixture-Model Q-Functions for Policy Iteration in Reinforcement Learning
arXiv:2512. 18763v2 Announce Type: replace Abstract: Unlike their conventional use as estimators of probability density functions in reinforcement learning (RL), this paper introduces a novel function-approximation role for Gaussian mixture models (GMMs) as direct surrogates for Q-function losses.
arXiv AI
Jul 24
Expert Behavior Prior Reinforcement Learning
arXiv:2607. 21302v1 Announce Type: new Abstract: Behavior prior reinforcement learning (BPRL) has emerged as a promising paradigm to improve sample efficiency in online reinforcement learning (RL) by leveraging policy priors derived from offline demonstrations.
arXiv Machine Learning
Sep 10
Decision-Centered Abstractions via Orthogonal Estimation of Difference-of-Q Functions
The paper introduces state abstractions that preserve the difference of Q‑functions for offline reinforcement learning, aiming to exclude irrelevant dynamics from rich state data. It proposes a dynamic generalization of the R‑learner that uses orthogonal estimation and sparse learning to estimate the Q‑function contrast, achieving faster convergence and consistency under a margin condition. Experiments on simulated and simulator‑augmented real data show variance reductions and demonstrate that the necessary information for sequential decision‑making can be smaller than that required for full state prediction.