Simulink Reinforcement learning environment, rate transition block applied on done signal preventing episode termination

Hello so I'm running a simulink environment and I have rate transition block between my agent and the state,reward,done signal.

The issue I think is happening is that my environment hits a episode terminating I defined in simulink, but when the done signal is activiated, due to the rate transition block since my agent sample time is slower then the enviroment sample time it doesn't get to the agent and the episode doesn't terminate. Then the next time my agent sample time hits, my environment is no longer in the terminating condition so the done signal goes back to false, the false done signal is fed into my agent and the epsiode essentially skipped termination. I have the scope of my done signal below.

Is there a way I can use a workspace variable in Matlab as my done signal, so basically I have done variable in matlab workspace and when I reach the terminating in simulink, I change that variable to true so that next time my agent sample time hits the epsiode terminates and in the reset function I reset that done signal back to false. Is this a viable solution?

Thanks

[Vincent]

@Drew Davis I'm really sorry for spamming again, but I would appreciate some advice.

[Vincent]

Ok now I'm not sure about the rate transition block being the cause. Basically I have a contact signal between 2 object that I use as my done signal, so when the objects collide the contact signal reads 1 and the done signal reads 1 but for some reason the episode doesn't reset.

Answer 1

[Deepak]

Hi Vincent, 

This is my understanding of the question.

Due to rate transition block and mismatched sample times between the agent and the environment, the agent sometimes makes a mistake of missing done = 1, wherein the done = 1 is used to ensure episode termination. So, the question is to know whether MATLAB workspace variable can be used as a done signal in Simulink to reliably terminate reinforcement learning episodes and reset it in the agent’s reset function.

To answer the question, yes, MATLAB workspace variable used as a persistent done flag can be a good workaround. Implementation could be as follows:

1. Setting the variable when termination is triggered
2. Read the variable in the agent logic
3. Reset the flag after termination is processed

There are chances of the cause being because of rate transition block. To ensure if that is the actual reason, confirm whether the done = 1 until the agent acknowledges and resets it. Check the reset condition and verify sample times of all relevant blocks. Kindly go through the MathWorks documentation attached below, which explains how to configure is-done signals and termination logics:

https://www.mathworks.com/help/reinforcement-learning/ug/create-custom-simulink-environments.html

Hope this solves the issue.