Following results and code are the implementation of simplest actor-critic in Gymnasium's Cart Pole environment. More actor-critic alorithms will be added in the learning of OpenAi Sunning Up tutorial.

RESULTS:

The simplest actor-critic algorithm takes too many steps to converge, it may be caused by large variance in sampling. If a baseline is reduced when updating policy, which refers to the trick used in?A2C, this phenomenon may be alleviated.

Visualizations of (i) changes in score?and?value approximation loss, and (ii) animation results.

CODE:

NetWork.py

QACAgent.py

train_and_test.py

The above code are mainly based on?Lesson 7 of the David Silver's lecture [1],?Chapter 10 of Shiyu Zhao's Mathematical Foundation of Reinforcement Learning [2], and?Chapter 10 of Hands-on Reinforcement Learning?[3].

Reference

[1] https://www.davidsilver.uk/teaching/

[2] https://github.com/MathFoundationRL/Book-Mathmatical-Foundation-of-Reinforcement-Learning

[3]?https://hrl.boyuai.com/