What is in a reinforcement learning saved agent .mat file

I am working with reinforcement learning in Simulink and MATLAB to perform continuous feedback control of a dynamic system. The saved agent files are very large after training on my model, and I was wondering what information is actually being saved in this file. Is there a way of looking at what is actually saved in the saved agent .mat file after training?

Answer 1

[Emmanouil Tzorakoleftherakis]

Why don't you load the file and check? When you saved the agen tin the .mat file, did you save anything else with it? Are you maybe saving a large experience buffer with the agent?

Another thing to consider is that along with the agent, some training data is also saved to be able to recreate training results post training. This additional data also contains any logged signals from Simulink/Simscape. I would recommend turning off signal logging and checking the size of the mat file again.

[Michael Ciriacy]

Thank you for your answer on my question.

I have set the experience buffer to a reduced length of 1000 and when we run the training the size of the saved agent grows very quickly with every episode. I have attached an image of this called “saved agents with reduced buffer”. With this setting the saved agent is still growing by about 20 Mb every episode.

I tried setting SaveExperienceBufferWithAgent to false and the saved agent file is still very large. An image of the saved agents with this setting is labeled “saved agents with no buffer”. The agent options that I used for this are below. With this setting the saved agent is still growing by about 20 Mb per episode.

agentOptions = rlTD3AgentOptions;
agentOptions.SampleTime = Ts;
agentOptions.DiscountFactor = 0.99;
agentOptions.MiniBatchSize = 256;
agentOptions.ExperienceBufferLength = 1e3;
agentOptions.TargetSmoothFactor = 5e-3;
agentOptions.TargetPolicySmoothModel.Variance = 0.2; % target policy noise
agentOptions.TargetPolicySmoothModel.LowerLimit = -0.5;
agentOptions.TargetPolicySmoothModel.UpperLimit = 0.5;
agentOptions.ExplorationModel = rl.option.OrnsteinUhlenbeckActionNoise; % set up OU noise as exploration noise (default is Gaussian for rlTD3AgentOptions)
agentOptions.ExplorationModel.MeanAttractionConstant = 1;
agentOptions.ExplorationModel.Variance = 0.1;
agentOptions.SaveExperienceBufferWithAgent = false;  % This was added to shut off the buffer

Thank you for your help with this question. Let me know if there is anything else you need.

[Emmanouil Tzorakoleftherakis]

Another thing to consider is that along with the agent, some training data is also saved to be able to recreate training results post training. This additional data also contains any logged signals from Simulink/Simscape. I would recommend turning off signal logging and checking the size of the mat file again.

[Michael Ciriacy]

After shutting off the signal logging in the Simulink model the size of the agent is still increasing about 16,000 KB per episode. This did help but is there anything else that could help decrease the size of the saved agent files? I would also like to know what else is being saved in the agent file at the end of the episode if that is possible. I am working with a model that could take a large amount of episodes so I would like to keep the saved agent as small as possible for a large number of episodes.

[Emmanouil Tzorakoleftherakis]

How big are the neural networks you are using?