Train a DQN Agent: RL Simulink with Simscape (Error: Discontinuities detected within algebraic loop)

Hello,

I am trying to replace the PI-Controller (the highlighted area with PWM Generator of 50 kHz) for the Buck-Converter sketch with A DQN-Agent!

Here the Simulink Enviroment with the PI-Controller, which is performing well

I tried to replicate the MATLAB example "Water Tank Model" in Simulink using RL, but using DQN instead of DDPG (since the action is discrete) and replaced the environment with the buck-converter with some other minor changes (tolerance ), but I am getting various errors, when I use validateEnvironment. Here my Matlab code with the simulink environment:

Obs_Info = rlNumericSpec([1 1]);
Obs_Info.Name = 'observations'; % measured current
numObservations = Obs_Info.Dimension(1);

% specifies discrete action specifications
Act_Info = rlFiniteSetSpec([0 1]);
Act_Info.Name = 'PWM';
numActions = Act_Info.Dimension(1);

env = rlSimulinkEnv('rl_converter','rl_converter/RL Agent', Obs_Info, Act_Info);

workspace = 'Stromsteuerung_rl';
env.ResetFcn = @(in)setVariable(in,'observations',0,'Workspace',workspace);

% the agent gets executed every SampleTime seconds of simulation time

Ts = 1/50000;        % (50 kHz)
Tf = (1/50000)*60;   % simulation time

%% Deep Neural Network
dnn = [
    featureInputLayer(1,'Normalization','none','Name','State')
    fullyConnectedLayer(40,'Name','CriticStateFC1')
    reluLayer('Name','CriticRelu1')
    fullyConnectedLayer(40, 'Name','CriticStateFC2')
    reluLayer('Name','CriticCommonRelu')
    fullyConnectedLayer(2,'Name','Action')];

%% Representation of Q-Values
Critic_Opts = rlRepresentationOptions;
Critic_Opts.LearnRate = 0.001;
Critic_Opts.GradientThreshold = 1;

critic = rlQValueRepresentation(dnn, Obs_Info, Act_Info, 'Observation',{'State'},Critic_Opts);

%% Agent options
Ag_Opts = rlDQNAgentOptions;

Ag_Opts.UseDoubleDQN = true;
Ag_Opts.TargetSmoothFactor = 1;
Ag_Opts.TargetUpdateFrequency = 4;
Ag_Opts.ExperienceBufferLength = 100000;
Ag_Opts.DiscountFactor = 0.9;
Ag_Opts.MiniBatchSize = 64;
Ag_Opts.SampleTime = Ts;

Ag_Opts.EpsilonGreedyExploration.Epsilon = 0.5;

agent = rlDQNAgent(critic,Ag_Opts);

%% Training options

Train_Opts = rlTrainingOptions;

Train_Opts.MaxEpisodes = 100;
Train_Opts.MaxStepsPerEpisode = ceil(Tf/Ts);
Train_Opts.StopTrainingCriteria = "AverageReward"; % or AverageSteps
Train_Opts.StopTrainingValue = 800; 
Train_Opts.Verbose = false;
Train_Opts.Plots = "training-progress";

%% Validate

validateEnvironment(env)

Warning: Discontinuities detected within algebraic loop(s), may have trouble solving

Warning: Convergence problem when solving algebraic loop containing 'rl_converter/stop simulation/Compare To Constant1/Compare'

at time 0.0. Simulink will try to solve this loop using Simulink 3 (R11) strategy.

Use feature('ModeIterationsInAlgLoops',0) to disable the strategy introduced in Simulink 4 (R12)

Why isn't this warning showing in the Water Tank Model and only in this one? Is Simscape not compatible with this application. If so, how could I change my environment to something that goes well with the RL Toolbox?

Any help is very much appreciated!

[Francisco Serra]

Hey! Could you solve it?

Answer 1

[MULI]

Hello,

I understand that you are trying to replace a PI controller with a DQN agent for your Buck Converter model and encountering issues with algebraic loops likely due to the usage of fast switching elements like the PWM.

In contrast, the Water Tank Model example doesn’t involve such fast dynamics or Simscape components, so it runs without these warnings.

You can follow the below suggestions to resolve this issue:

• You could try breaking the algebraic loop by adding a small delay (e.g.,”Unit Delay” block) between the controller output and the feedback signal.
• Since you are using DQN, a discrete-action RL algorithm, ensure that your environment runs with a discrete sample time. If the continuous-time nature of Simscape is causing issues, try setting discrete sample times in all parts of your model

For any additional insights on how to create and simulate a reinforcement learning (RL) environment using both MATLAB and Simulink you can refer below link:

https://www.mathworks.com/help/reinforcement-learning/ug/simulate-agent-and-environment-in-matlab-and-simulink.html