Doubts in developing Simulink model for Reinforcement learning

Hello, I am new to the RL in matlab. From the onramp course and online sources I understood that I should have a simulink environment with an RL agent to use this. My system is described by a set of differential equation. How can I vary the initial condition in the integrator during training? Also it is the initial condition for output, but how to set the initial condition for input? Currently I developed a Simulink model with integrator block having output's initial condition to be zero, but for input I did not specify. I tried to train a DQN agent and I observed that it tracks the setpoint reasonably, however, when I plotted the input values given by RL agent, it started from the point 50 and then evolved with time, however I did not specify this value. Similarly, for a different output condition of 12 in the integrator block, the input value from RL agent block started from 10, this value of 10 also not specified by me. Can you please help me with two things.

1. How to give the initial condition for output externally to the integrator block during training?
2. How to specify the initial condition of input value corresponding to the output initial condition? (I don't want it to be randomly considered).

[K.R]

Thank you, I will try these things

Answer 1

[Kothuri]

Hi K.R,

In Reinforcement Learning (RL) within MATLAB and Simulink environments, it is crucial to manage the initial conditions for both inputs and outputs for training agents effectively.

To give the initial condition for output externally for the integrator block, you can use the External option in the block's parameters under Initial condition source.

• Double-click on Integrator block
• Set Initial condition source to external.
• This exposes an additional input port on the Integrator block for the initial condition.

You can refer the below Documentation link for more info on Integrator

https://www.mathworks.com/help/releases/R2022b/simulink/slref/integrator.html?searchHighlight=integrator&s_tid=doc_srchtitle#f7-890704_sep_f7-1513452

For varying conditions during training, you can use a mechanism that updates the value based on the training episode. A MATLAB Function block can be programmed to output different values based on a global variable or a parameter that you update.

For specifying initial conditions for inputs, which correspond to the output initial conditions, you will likely need to create a custom setup where the initial input is a function of the desired output or state. This can be done by using a MATLAB Function block to define a relationship between the desired output initial condition and the initial input condition. This relationship should be based on your system's dynamics and the specific behaviour you're aiming to achieve with the RL agent. Or to implement logic that calculates the initial input based on the desired output condition. This could be a direct formula, a lookup table, or more complex logic.

You can refer the below Documentation link for more info on MATLAB Function block

https://www.mathworks.com/help/releases/R2022b/simulink/slref/matlabfunction.html?&s_tid=doc_srchtitle

Hope this helps