Hi, I'm using Reinforcement Learning on a control problem, specifically a TD3 agent. I have an order 3 plant and I want to use RL to find the optimal values for PI gains, so i'm basing on this matlab link. My problem is very similar to the matlab example, but instead of water tank I have to control the input airflow to generate a temperature signal that follows this reference:
Summarizing:
- action: airflow speed (%)
- Observation: error (reference temperature vs real temperature) and integral error
- Reward function:
- +10 if the error < 0.1, -1 otherwise
- -1000 if temp < 0 (episode stop condition)
- -10 if action < 50 (this helps to avoid bad states)
I have 3 neural networks:
- Actor: a single neuron. The weights are the PI gains.
- Critics: TD3 algorithm uses 2 critics, they have the same architecture
This is one of the best agent I've trained so far:
- Learning rate: actor: 0.001, critic: 0.01
I trained this agent 1000 episodes more, but I get worse simulation results.
Looking at the action signal generated by this RL agent, it's fairly good, in control terms I think.
The problem here, it's Episode q0. According to matlab:
For agents with a critic, Episode Q0 is the estimate of the discounted long-term reward at the start of each episode, given the initial observation of the environment. As training progresses, if the critic is well designed. Episode Q0 approaches the true discounted long-term reward, as shown in the preceding figure.
Episode q0 (yellow line) doesn't approach the episode reward (blue line) and average reward (red line). So, according to this, my agent is very bad, right? but why I'm getting good results? And also, how can I fix this? Just trying another critic architecture like more layers?
