Reinforcement learning: "NextObs" vs. "LoggedState" in step function

Hi,

I could not find out what the difference between "NextObs" and "LoggedSignals" is in the step function. In all scripts both are passed on from the step function.

[NextObs,Reward,IsDone,LoggedSignals] = myStepFunction(Action,LoggedSignals)

"LoggedSignals" is obviously used for the next step, but what is "NextObs" used for?

Thanks!

Answer 1

[Emmanouil Tzorakoleftherakis]

Actually, NextObs is the important thing here. It represents the value of your states after you apply current action and integrating one step.

LoggedSignals is where you can log information to view later - can be left empty too.

[Anne Tscheliessnig]

Taking the example of the Pendulum on https://de.mathworks.com/help/reinforcement-learning/ug/create-custom-reinforcement-learning-environment-in-matlab.html :

The myStepFunction is taking NextObs, Reward, IsDone and LoggedSignal and when calling needs Action and LoggedSignals

function [NextObs,Reward,IsDone,LoggedSignals] = myStepFunction(Action,LoggedSignals)

Then I would need LoggedSignals for the next step? And NextObs is being used for the agent? In this case I could not leave LoggedSignals empty, or?

This example is very confusing for me, because actually it states almost to the end in the myResetFunction

NextObs = LoggedSignals.State;

[Emmanouil Tzorakoleftherakis]

Oh you were looking at creating custom environments with functions - I was looking at creating environments with classes by running e.g.

rlCreateEnvTemplate('myenv')

where LoggedSignals is not that important since you can use class variables to store the states.

I suspect the reason you need both LoggedSignals and NextObs is to create a unified way of using custom environments regardless of how you create it. NextObs is probably what the agent is using when interacting with the environment, whereas LoggedSignals is a way to save intermmediate values if you don't use classes to create your custom env.

[lfyx]

Hello, may I ask that, can the "sim" function output the LoggedSignals to the work space? Many information about the simulation action or observarion are saved in the LoggedSignals. However, the output of "sim" is the experince structure.

[Maha Mosalam]

Hi, what about the xact role of IsDone flag it it shuld be true or false or what?