it feels like my algorithm never starts exploiting and is just exploring throughout the training how can i reduce the exploration?
PPO agent low reward episodes
25 vues (au cours des 30 derniers jours)
Afficher commentaires plus anciens
I am trying to implement PPO agent and i m getting rewards as shown and i have tried tuning hyperparameter settings but still training looks like this and I dont know what is the isssue I also tried to use Isdone signal to terminatee the episode when reward reaches below a certain value but still no use . can someone help.?
![](https://www.mathworks.com/matlabcentral/answers/uploaded_files/1565219/image.png)
Réponses (1)
Shivansh
le 27 Déc 2023
Hi Sourabh,
I understand that you are training a PPO agent which is stuck in exploration and not able to learn a stable policy. This is resulting in poor training as shown in the provided graph.
Here are a few steps to improve the performance of your model:
- Try normalizing or scaling down the rewards to a smaller range.
- A PPO agent uses entropy regularization to encourage exploration. Try reducing the entropy regularization coefficient gradually and observe the results.
- You can try verifying your “IsDone” flag’s implementation. It can terminate the episode at the right time to prevent the agent from learning undesired states.
Apart from the above points, you can also try experimenting with learning rates, hyperparameter tuning, and network architecture, etc to find the best configurations for your problem.
You can use a known benchmark environment to verify that your PPO implementation works as expected. Once you have confirmed that the agent can learn in a simpler setting, gradually reintroduce complexity.
You can refer to the common lander vehicle example to learn more about the working of a PPO agent in MATLAB: https://in.mathworks.com/help/reinforcement-learning/ug/train-ppo-agent-to-land-vehicle.html.
For more information on PPO agents, refer to the following documentation: https://in.mathworks.com/help/reinforcement-learning/ug/ppo-agents.html.
If the issue persists, provide more information related to problem statement and the environment. It will help to get a better understanding of the issue you are facing.
Hope it helps!
3 commentaires
Shivansh
le 3 Jan 2024
Hi Sourabh!
You don't have to change the structure of reward function to normalize rewards. You can use reward normalization techniques like dividing by a maximum possible reward or using statistical normalization (subtracting the mean and dividing by the standard deviation of the rewards).
Adding a large positive constant might not be a good way as this might lead to suboptimal policies. If adding a constant improves learning, there might be a case that the original rewards might be too negative or sparse. You can try revising your reward function to better balance positive and negative rewards, guiding the agent more effectively towards the optimal policy.
Hope it helps!
Voir également
Catégories
En savoir plus sur Environments dans Help Center et File Exchange
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!