arXiv Machine Learning By Jan Felix Kleuker, Koen Ponse, Aske Plaat, Thomas M. Moerland

Revisiting Critic Estimation in Soft Actor-Critic Algorithms

Read the original on arXiv Machine Learning →

The paper investigates alternative critic designs for the Soft Actor-Critic (SAC) algorithm, exploring learning soft advantages, multi-step return targets, and training multi-step critics from replay buffer data. Six critic variants are compared across fourteen tasks, revealing that training a λ-return critic on replayed trajectory segments yields the most performance gains, while learning soft advantages or using soft GAE offers no benefit. These findings suggest that uncorrected multi-step targets can be effectively applied to SAC and potentially other reinforcement learning algorithms.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

arXiv AI
Oct 9

A Geometric Approach to Soft Actor-Critic with Zonotopes for Locomotion Learning

The paper introduces GeZo‑SAC, a variant of Soft Actor‑Critic that incorporates geometric representations (zonotopes) to adapt critic pessimism based on state and action. Each critic predicts a zonotope whose width is subtracted from the critic value as a pessimistic offset, and the disagreement between critics is measured and used to blend the critics via a log‑sum‑exp aggregation. Experiments on MuJoCo‑v5 locomotion tasks show GeZo‑SAC achieving the highest mean return on Ant‑v5 and Hopper‑v5, while also reducing actuator work and action effort and maintaining near‑zero overestimation frequency.

By Panagiotis Roditis, Panagiotis P. Filntisis, Petros Maragos
arXiv AI
Sep 28

PolicyAttention: Softmax Attention Implements Policy Mirror Descent for Closed-Loop Control

The paper investigates whether causal softmax attention can realize policy mirror descent (PMD) as a repeated controller rather than a one‑step algebraic identity. It constructs a fixed causal‑softmax actor–environment–one‑step‑critic protocol, detailing actor, routing, sampling, and normalization residuals, and shows that a frozen one‑step audit model closely approximates PMD. Empirical results demonstrate that the learned actor with an exact one‑step critic achieves median policy loss only about 5% higher than the exact PMD oracle across multiple control settings.

By Yuhe Sui, Yingzhi Tang, Shufang Chen