DDPG Agent isn't learning (reward 0 for every episode)

Hello,

I suspect that the SampleTime Ts has something to do with it, I set Ts = 1e-6, but the trainnig is still going very fast.

I tried the same thing that was used for the Water Tank Model:

https://de.mathworks.com/help/reinforcement-learning/ug/create-simulink-environment-and-train-agent.html?searchHighlight=create%20simulink%20rl&s_tid=srchtitle

Just like in the example my system is very well controlled with a PI-Controller, but the DDPG Agent isn't learning anything.

What exactly is wrong with this system?

Answer 1

[Emmanouil Tzorakoleftherakis]

The reason why you see 0 rewards is because thw IsDone flag (which is used to terminate episodes early) is immediately set to true at the beginning of each episode. Either set it to false, or set the appropriate logic for you system.

Note that there are other things that need to be fixed in your setup, the most notable one being the agent sample time (which is currently very small) and the episode duration/max steps (which is currently very large). Unless you adjust these to more reasonable values for RL, your training will take days

[Reinforcement Learning]

Hey @Emmanouil Tzorakoleftherakis!

Thanks a lot for answering. This worked very good! Thank you for giving me this information. I have one issue still. The Action-Space I choose was discrete [0, 1] to replicate a PWM signal. This wasn't so accurate, since I am aiming at (50 kHz) and getting more even with setting Ts to (1/50k)s. Do you maybe know how I could change my action-space to have a better PWM?

[Emmanouil Tzorakoleftherakis]

Not sure - maybe using the duty cycle as an action would work? Regardless, it seems you need to rethink the inputs to your problem. Maybe instead of PWM, you should be setting the desired/average value.

[Reinforcement Learning]

I tried some discrete duty cycle value like [0 0.15 0.25 0.3 0.5] but that made it even worse. What do you mean by setting the desired/ average value? Isn't that an Input?

I'm actually just trying to stay discrete (DQN), otherwise I will have to switch to DDPG

[Emmanouil Tzorakoleftherakis]

Tha's what I am saying, you may need to reconsider your inputs/outputs. I am not sure what you mean by "made it even worse". If you are not seeing good training results, there could be a lot of other reasons including training options and hyperparameters, networks architectures etc. I would give the duty cycle idea another try and spend more time on the hyperparameters

[yovel atia]

Hi

I get the same error, only for me the graph converges to -5000 constant

I do not know how to fix it I would be very happy to help

thanks!