Deploy trained policy to simulink model

I am trying to deploy a trained policy of the reinforcement learning toolbox to a simulink model. This model has to be compatible with code generation (not just normal & accelerator mode). So using an Interpreted function block as stated here (https://www.mathworks.com/matlabcentral/answers/485994-reinforcement-learning-how-to-use-a-trained-policy-as-a-controller-block-in-simulink?s_tid=answers_rc1-1_p1_BOTH) is not suitable. But I am stuck at this point with several errors. Can please anyone provide an example on how to do this? Also the matlab documentation is laacking here, just showing how to compile a mex file of the evaluatePolicy function. Therefore, I run the follwing lines,

load('Agent3524.mat','saved_agent')
generatePolicyFunction(saved_agent)

and get my eveluatePolicy function,

function action1 = evaluatePolicy(observation1)
%#codegen

% Reinforcement Learning Toolbox
% Generated on: 20-Feb-2020 17:30:58

action1 = localEvaluate(observation1);
end
%% Local Functions
function action1 = localEvaluate(observation1)
persistent policy
if isempty(policy)
	policy = coder.loadDeepLearningNetwork('agentData.mat','policy');
end
action1 = predict(policy,observation1);
end

I have a simple Simulink model:

with the matlab function block code:

function y = fcn(u)

    y = evaluatePolicy(u);
end

Now I get a bunch of error messages, and I dont know how to solve them:

Undefined function or variable 'dltargets'.

P-code function 'DeepLearningNetwork.p' produced an error.
Component:MATLAB Function | Category:Coder error
Function call failed.

Function 'loadDeepLearningNetwork.m' (#31.3569.3643), line 100, column 15:
"coder.DeepLearningNetwork(coder.const(matfile), coder.const(''), param{:})"
Launch diagnostic report.
Component:MATLAB Function | Category:Coder error
Function call failed.

Function 'evaluatePolicy.m' (#29.286.341), line 13, column 11:
"coder.loadDeepLearningNetwork('agentData.mat','policy')"
Launch diagnostic report.
Component:MATLAB Function | Category:Coder error
Persistent variable 'policy' must be assigned before it is used. The only exception is a check using 'isempty(policy)' that can be performed prior to assignment.

Function 'evaluatePolicy.m' (#29.365.371), line 15, column 19:
"policy"
Launch diagnostic report.
Component:MATLAB Function | Category:Coder error
Function call failed.

Function 'evaluatePolicy.m' (#29.140.167), line 7, column 11:
"localEvaluate(observation1)"
Launch diagnostic report.
Component:MATLAB Function | Category:Coder error
Function call failed.

Function 'MATLAB Function' (#23.29.46), line 3, column 9:
"evaluatePolicy(u)"
Launch diagnostic report.
Component:MATLAB Function | Category:Coder error
Errors occurred during parsing of MATLAB function 'test/MATLAB Function'
Component:MATLAB Function | Category:Coder error
Simulink cannot determine sizes and/or types of the outputs for block 'test/MATLAB Function' due to errors in the block body, or limitations of the underlying analysis. The errors might be inaccurate. Fix the indicated errors, or explicitly specify sizes and/or types for all block outputs.
Component:MATLAB Function | Category:Coder error
Simulink cannot determine sizes and/or types of the outputs for block 'test/MATLAB Function' due to errors in the block body, or limitations of the underlying analysis. The errors might be inaccurate. Fix the indicated errors, or explicitly specify sizes and/or types for all block outputs.
Component:Simulink | Category:Model error
An error occurred while propagating data type 'double' through 'test/MATLAB Function'.
Component:Simulink | Category:Model error

Any help is appreciated! Thanks!

Answer 1

[Emmanouil Tzorakoleftherakis]

Hello,

Looks like the dimensions cannot be determined automatically. If you double click the MATLAB Fcn block and then click "Edit Data", you can specify the input and output dimensions of the block, i.e. for y and u. It would also be helpful to initialize y before calling evaluatePolicy with a vector that has the same dimensions as the policy output.

Just making sure, you also need to have installed the support package MATLAB Coder for Deep Learning and the mkl-dnn library.

Please let me know if you still have trouble after changing the above.

[SomeMatlabUser]

The error messages do change, but it still does not work. I did set the port dimensions to the correct values ( 4 in (u) and 1 out (y)), and also added a

double()

function around the evaluatePolicy function, since it seems to output single (float) valued data. Also I initialized y with 0.

My updated Matlab function block fcn:

function y = fcn(u)

    y = 0;
    y = double(evaluatePolicy(u));
end

Also I installed the package you linked in your answer. This does indeed change the first line in the error message, but does not tell me how to solve the error:

Undefined function or variable 'dltargets.cudnn'.

P-code function 'DeepLearningNetwork.p' produced an error.
Component:MATLAB Function | Category:Coder error
Function call failed.

Function 'loadDeepLearningNetwork.m' (#31.3569.3643), line 100, column 15:
"coder.DeepLearningNetwork(coder.const(matfile), coder.const(''), param{:})"
Launch diagnostic report.
Component:MATLAB Function | Category:Coder error
Function call failed.

Function 'evaluatePolicy.m' (#29.286.341), line 13, column 11:
"coder.loadDeepLearningNetwork('agentData.mat','policy')"
Launch diagnostic report.
Component:MATLAB Function | Category:Coder error
Persistent variable 'policy' must be assigned before it is used. The only exception is a check using 'isempty(policy)' that can be performed prior to assignment.

Function 'evaluatePolicy.m' (#29.365.371), line 15, column 19:
"policy"
Launch diagnostic report.
Component:MATLAB Function | Category:Coder error
Function call failed.

Function 'evaluatePolicy.m' (#29.140.167), line 7, column 11:
"localEvaluate(observation1)"
Launch diagnostic report.
Component:MATLAB Function | Category:Coder error
Function call failed.

Function 'MATLAB Function' (#23.47.64), line 4, column 16:
"evaluatePolicy(u)"
Launch diagnostic report.
Component:MATLAB Function | Category:Coder error
Errors occurred during parsing of MATLAB function 'test/MATLAB Function'
Component:MATLAB Function | Category:Coder error
Errors occurred during parsing of MATLAB function 'test/MATLAB Function'
Component:Simulink | Category:Model error

[Emmanouil Tzorakoleftherakis]

I see. Just to give you an idea of what's happening when the MATLAB Fcn block is compiled (or when you generate C/C++ code from a deep neural network in MATLAB for that matter), you are using the MATLAB Coder for Deep Learning package to link the code generated to optimized target specific libraries that will do the inference part. As of R2019b, Intel and ARM processors are supported, and the corresponding 3rd party libraries that are linked to are the mkl-dnn library (for Intel processors) and ARM compute (for ARM).

So, from the error in the first line, it seems that either 3rd party library required is not linked or you have not installed/built it. If you look at this page, there is a link to the prerequisites needed for code generation from deep nets.

[SomeMatlabUser]

From what I understand, it sounds like I should definitly be possible. Seems like mkl-dnn is not installed correctly on my system. I will have a look into it.

But just to be sure: Is it technically possible to solve my problem and does this just depend on my pc or does there exist a limitation inside Matlab/Simulink, that it is just not possible to generate code form a simulink model with an policy from the RL-toolbox?

[Emmanouil Tzorakoleftherakis]

The steps I mention above are for running simulations with a MATLAB Fcn block (without calling the interpreter) that does inference on a neural network created by Reinforcement Learning Toolbox (or Deep Learning Toolbox). If you also want to generate readable C/C++ code from the model, there is one more step which involves replacing the MATLAB Fcn block with an s-function. This video shows how to do this for GPU code, and we are working on one that does the same with C/C++ code using MATLAB Coder from Reinforcement Learning Toolbox. With s-functions you should be able to generate code from your model assuming a) all the layers you are using support code generation (see here) b) you don't have multiple input layers, i.e., you are generating code from an actor and not from a critic.

The development team is actually working on a more user-friendly way to bring neural networks into Simulink, so think of the s-function approach as a temporary workaround. I hope this helps.

[Sviatoslav Klos]

Hello.

I would like to ask related question.

Emmanouil Tzorakoleftherakis, would you also working on capability of code generation from a critic? Will it be in 2020a?

Since in documentation of the documentation here (https://www.mathworks.com/help/reinforcement-learning/ref/rl.agent.rldqnagent.generatepolicyfunction.html) it is the example of DQN agent policy generation that up to me is not possible for now.

[Emmanouil Tzorakoleftherakis]

Yes the development team is actively working on addressing this limitation.