What exactly is Episode Q0? What information is it giving?

Reading documentation I find that "For agents with a critic, Episode Q0 is the estimate of the discounted long-term reward at the start of each episode, given the initial observation of the environment. As training progresses, if the critic is well designed. Episode Q0 approaches the true discounted long-term reward"

But I cannot grasp exactly what is Q0 because, except in a few examples (like this one) where it "converges" to some value rather quickly, I have seen Q0 value do different things and I cannot understad or interpret them (like the two examples shown here). I also don't understand what "true discounted reward" means exaclty. Is it for each episode, average or something cumulative?

In this answer it is suggested that Q0 should track the average episode reward, but I don't see that in the examples.

For example, in the cartpole example, if one continues the training for more episodes (changing the stop training criteria to avoid stopping for average reward), Q0 value reaches very high values that have nothing to do with the average reward or the episodes. I simulated 1000 episodes for the cartpole example and Q0 values even mess up the scale because they go way too high. The agent seams too learn properly and it even manages to get out of some local minimums sucessfully but still, I cannot grasp what information Q0 yields

I have not found Q0 defined in Reinforcement Learning bibliography either. Could you please clarify a bit or give some bibliogtaphy where I can read further on this specific parameter?

Answer 1

[Emmanouil Tzorakoleftherakis]

Q0 is calculated by performing inference on the critic at the beginning of each episode. Effectively, it is a metric that tells you how well the critic has been trained. If you had the perfect critic that could accurately predict the expected long term reward based on the current observation at the beginning of the episode, this value should overlap with the actual total reward collected during that same episode.

In general, it is not required for this to happen for actor-critic mathods. The actor may converge first and at that point it would be totally fine to stop training.

Hope that helps

[Cecilia S.]

Oh, I think I am even more confused now :/

So, in the case of Q and DQN which don't have actors, we definitely need Q0 value to be as close to our individual episode reward as possible to assume it has converged?

[Emmanouil Tzorakoleftherakis]

Ideally they should be close yes. Here are some examples where training takes a while to converge and this behavior is observed:

https://www.mathworks.com/help/reinforcement-learning/ug/train-dqn-agent-for-lane-keeping-assist.html

https://www.mathworks.com/help/reinforcement-learning/ug/train-dqn-agent-for-lane-keeping-assist-using-parallel-computing.html

[Maha Mosalam]

how matlab calculate the epicode reward and average reward ? if the epicoe reward isthe summ to all rewards during all epicodes steps? I want to know the difference??

[Florian Rosner]

The average reward is default calculated over 5 Episodes you can change it e.g. to 10 by setting

rlTrainingsOptions.ScoreAveragingWindowLength = 10;

The reward is calculated by the sum of all steps inside an episode regarding my understanding.

In order to get back to the original question. How is the Episode Q0 exactly calculated?

And can the difference between Q0 and EpisodeReward be interpreted as the loss function mentioned in the example?

https://de.mathworks.com/help/reinforcement-learning/ug/train-reinforcement-learning-policy-using-custom-training.html#TrainRLPolicyUsingCustomTrainLoopExample-4

[蔷蔷 汪]

That is to say, when using ddpg algorithm, does episodeq0 and episode reward not need to coincide？

[Arman Ali]

Same question, anyone who can show Episode Q0 calculation?

[轩]

I think Q0 is still useful in Actor-Critic methods. Can I take it this way, in the AC algorithm if Q0 does not converge with average reward, that means critic's converging rate is slower than actor, we should set a bigger learning rate or something else.

[Jayandi]

Thank you Emmanouil. I think your definition is correct.

Converging of Q0 and reward average is the best a training could achieve but my case experience it is not necessary. So this is my conclusions:

1. Q0 should be almost stable, meaning it is around a number, for insant -7.xxxx. Perhaps at beginning it is not but after several epoch it arrives on a stable number. Q0 which is going upward and downward, not stable in a number, perhaps required several adjusment in hyperparameters, networks, or else.
2. A stable Q0 and a stable average rewards, for my case, is enough to say the training is converged. It is proven while testing of the agent afterward.
3. Of course, it is always tempting to make changes as far as possible to make the avg rewards and Q0 converge, but I don't have luck yet. If anyone could share the recipe, I would glad to hear.
4. In short, Q0 is helping to check our training conditions.