Hi Xiang,
It is my understanding that the model presented in the code is designed for a Proximal Policy Optimization (PPO) algorithm, incorporating both a critic and an actor network to handle environments with temporal dependencies. The critic network features a sequence input layer, multiple fully connected layers with tanh activations, an LSTM layer for capturing temporal patterns, and a final fully connected layer for value function output. The actor network follows a similar initial structure, with an LSTM layer to process sequences, but then diverges into two paths: one determining the mean and the other the standard deviation of the action distribution, using fully connected layers followed by ‘tanh’ and ‘softplus’ activations, respectively. This bifurcation allows the actor to output a continuous Gaussian action distribution, informed by both the central tendency and variability of actions, based on the current state observations.
Regarding the hyperparameters, the discount factor (0.992) is quite high, and is appropriate for tasks where long term benefits are significant. The mini batch size can be larger to reduce variance but may require more memory and computational power.
Depending on your specific application, you might find better performance with different architectures. For example, additional LSTM layers or different sizes for the fully connected layers might capture the policy or value function more effectively. Normalizing inputs to the network can significantly improve training stability and performance. Also, regularly evaluating your agent in the environment using a separate set of episodes from training to monitor for overfitting or underfitting.
Ultimately, the "reasonableness" of your setup can only be fully assessed through experimentation and iterative modification based on the performance of your agent in the environment it is being trained in.
Hope this helps.
