Multi agent reinforcement learning for gain tuning power electronics

i am trying to setup environment and agents for the DRL, the env setupt code is as follow .mlx

% Load the Simulink model

mdl = 'RL_two_agents.slx';

open_system(mdl);

% I/O specifications for agent A

obsInfo1 = rlNumericSpec([3, 1]);

obsInfo1.Name = 'observations1';

obsInfo1.Description = 'a, b, c'; % Add space after commas for clarity

actInfo1 = rlNumericSpec([1 ,1], 'LowerLimit', 1, 'UpperLimit', 20);

actInfo1.Name = 'gain_1';

% I/O specifications for agent B

obsInfo2 = rlNumericSpec([3 ,1]);

obsInfo2.Name = 'observations2';

obsInfo2.Description = 'a2, b2, c2'; % Add space after commas for clarity

actInfo2 = rlNumericSpec([1 ,1], 'LowerLimit', 0.01, 'UpperLimit', 1);

actInfo2.Name = 'gain_2'; % Add action name for clarity

% Combine observation and action specs into cell arrays

observationInfo = {obsInfo1,obsInfo2}; % Cell array of observation info

actionInfo = {actInfo1,actInfo2}; % Cell array of action info

% Create the reinforcement learning environment

env = rlSimulinkEnv('RL_two_agents', ...

'RL_two_agents/RL Agent1', ...

'RL_two_agents/RL Agent2', ...

observationInfo,actionInfo); % Explicitly pass as cell arrays

% Set the reset function

env.ResetFcn = @(in)localResetFcn(in);

the errror msg: The error message indicates that the observationInfo and actionInfo parameters should be either rl.util.RLDataSpec objects or cell arrays of rl.util.RLDataSpec objects.

when i check the class; disp(class(obsInfo2));rl.util.rlNumericSpec

what step should i take to resolve this issue.

[Aravind]

Hi @Muhammad,

Could you please share your Simulink model file and other related files? This will help in identifying and debugging the issue.

[Muhammad]

@Aravind Unfortunately, I'm unable to share the model files at the moment, but if you have any questions about the model that could help resolve this issue, I’d be glad to provide further details. I'd also like to mention that the setup works fine for a single agent with two actions. However, when I attempt to use two agents with separate actions within the same environment, I encounter this error.

Both the agents have different observations and actions as well as the reward function is customized for both the agents

Answer 1

[Umar]

Hi @Muhammad,

Upon reviewing the code, it appears that the specifications for observations and actions are created using rlNumericSpec, which is indeed a valid class for defining observation and action spaces. However, the error message indicates a mismatch in the expected input types. The issue may arise from the way the observationInfo and actionInfo are being passed to the rlSimulinkEnv function. To resolve this issue, ensure that the observationInfo and actionInfo are explicitly defined as cell arrays of rl.util.RLDataSpec objects. The current code already does this correctly, but it is essential to verify that the rlNumericSpec objects are indeed being recognized as rl.util.RLDataSpec objects.Here is a revised version of the code with additional checks:

% Load the Simulink model
mdl = 'RL_two_agents.slx';
open_system(mdl);

% I/O specifications for agent A
obsInfo1 = rlNumericSpec([3, 1]);
obsInfo1.Name = 'observations1';
obsInfo1.Description = 'a, b, c'; % Add space after commas for clarity
actInfo1 = rlNumericSpec([1, 1], 'LowerLimit', 1, 'UpperLimit', 20);
actInfo1.Name = 'gain_1'; 

% I/O specifications for agent B
obsInfo2 = rlNumericSpec([3, 1]);
obsInfo2.Name = 'observations2';
obsInfo2.Description = 'a2, b2, c2'; % Add space after commas for clarity
actInfo2 = rlNumericSpec([1, 1], 'LowerLimit', 0.01, 'UpperLimit', 1);
actInfo2.Name = 'gain_2'; 

% Combine observation and action specs into cell arrays
observationInfo = {obsInfo1, obsInfo2}; % Cell array of observation info
actionInfo = {actInfo1, actInfo2}; % Cell array of action info

% Create the reinforcement learning environment
env = rlSimulinkEnv(mdl, ...
  {'RL_two_agents/RL Agent1', 'RL_two_agents/RL Agent2'}, ...
  observationInfo, actionInfo); % Explicitly pass as cell arrays

% Set the reset function
env.ResetFcn = @(in)localResetFcn(in);

Make sure that the agent block paths are passed as a cell array in the rlSimulinkEnv function and verify that the observationInfo and actionInfo are indeed recognized as cell arrays of rl.util.RLDataSpec objects.

Hope this helps.

[Umar]

Hi @Muhammad,

Did you make any progress yet. Please let me know.

[Muhammad]

hey @Umar following your suggestion i got this error

Error using rlSimulinkEnv (line 67)

The value of 'agentBlock' is invalid. Expected input to be one of these types:

char, string

[Umar]

Hi @Muhammad,

Since you have not shared your code, I will address address the error you kept experiencing with the rlSimulinkEnv`function, let me break down this time the key areas of concern. The error message you received—“The value of agentBlock is invalid. Expected input to be one of these types: char, string suggests that the paths provided for your agent blocks may not be formatted correctly. This typically happens if they are passed as a cell array or another data structure that is not recognized as a character array or string. So, go back to your code and make sure that the agentBlocks argument in rlSimulinkEnv is passed correctly. If you have multiple agents, they should be passed as a string array or a character vector, not as a cell array. Here is how you can adjust your code:

   agentBlocks = ["RL_two_agents/RL Agent1", "RL_two_agents/RL Agent2"]; 
   % String array
   env = rlSimulinkEnv(mdl, agentBlocks, observationInfo, actionInfo);

While your observation and action specifications appear correct, it is prudent to double-check that they are indeed being recognized as rl.util.RLDataSpec objects. You can verify this by checking their class:

   class(observationInfo{1}) % Should return 'rl.util.RLDataSpec'
   class(actionInfo{1}) % Should return 'rl.util.RLDataSpec'

Make sure that the Simulink model (`mdl`) is loaded correctly and contains the specified agent blocks. Check if there are any typos in the block names or paths. Use MATLAB's built-in functions like get_param() to confirm that your agent blocks exist in the model and are accessible.Here is again how your adjusted code snippet might look:

   mdl = 'RL_two_agents.slx';
   open_system(mdl);

   % I/O specifications for agent A
   obsInfo1 = rlNumericSpec([3, 1]);
   obsInfo1.Name = 'observations1';
   obsInfo1.Description = 'a, b, c'; 

   actInfo1 = rlNumericSpec([1, 1], 'LowerLimit', 1, 'UpperLimit', 20);
   actInfo1.Name = 'gain_1'; 

   % I/O specifications for agent B
   obsInfo2 = rlNumericSpec([3, 1]);
   obsInfo2.Name = 'observations2';
   obsInfo2.Description = 'a2, b2, c2'; 

   actInfo2 = rlNumericSpec([1, 1], 'LowerLimit', 0.01, 'UpperLimit', 1);
   actInfo2.Name = 'gain_2'; 

   % Combine into cell arrays
   observationInfo = {obsInfo1, obsInfo2}; 
   actionInfo = {actInfo1, actInfo2}; 

   % Create environment
   agentBlocks = ["RL_two_agents/RL Agent1", "RL_two_agents/RL Agent2"];
   env = rlSimulinkEnv(mdl, agentBlocks, observationInfo, actionInfo); 

   % Set reset function
   env.ResetFcn = @(in)localResetFcn(in);

A piece of advice, consider adding try-catch blocks around your environment creation code to capture and handle errors more gracefully.

By ensuring that all parameters are correctly formatted and checking for typographical errors or misconfigurations within your Simulink model setup, you should be able to resolve this issue effectively.

If further errors arise after making these adjustments, feel free to share those messages for additional troubleshooting assistance.

[Muhammad]

with this line of code the error is resolved

blks = mdl + ["/RL Agent1", "/RL Agent2"];

env = rlSimulinkEnv(mdl,blks,observationInfo,actionInfo);

but in the Reinforcement designer app when i try to add new agent it gives this error

Error in executing callback registered with ViewModel:

Error using matlab.ui.control.internal.model.AbstractStateComponent/set.Items (line 204)

'Items' must be a 1-by-N cell array of character vectors or a string array.

[Umar]

Hi @Muhammad,

Glad to know that you are making progress. You mentioned that you resolved the initial error by using:

blks = mdl + ["/RL Agent1", "/RL Agent2"];
env = rlSimulinkEnv(mdl, blks, observationInfo, actionInfo);

This is a valid approach, but ensure that blks is indeed a string array. You can explicitly define it as follows:

blks = string(mdl + ["/RL Agent1", "/RL Agent2"]); 
% Ensure blks is a string array

Please still make sure that your observation and action specifications are correctly defined as rl.util.RLDataSpec objects. Here’s how you can define them:

% I/O specifications for agent A
obsInfo1 = rlNumericSpec([3, 1]);
obsInfo1.Name = 'observations1';
obsInfo1.Description = 'a, b, c'; 
actInfo1 = rlNumericSpec([1, 1], 'LowerLimit', 1, 'UpperLimit',   20);
actInfo1.Name = 'gain_1'; 

% I/O specifications for agent B
obsInfo2 = rlNumericSpec([3, 1]);
obsInfo2.Name = 'observations2';
obsInfo2.Description = 'a2, b2, c2'; 
actInfo2 = rlNumericSpec([1, 1], 'LowerLimit', 0.01, 
'UpperLimit', 1);
actInfo2.Name = 'gain_2'; 

% Combine into cell arrays
observationInfo = {obsInfo1, obsInfo2}; 
actionInfo = {actInfo1, actInfo2}; 

With the corrected agent blocks and specifications, you can create the environment as follows:

mdl = 'RL_two_agents.slx';
open_system(mdl); % Ensure the model is open

% Create environment
agentBlocks = string(mdl + ["/RL Agent1", "/RL Agent2"]); 
% Ensure this is a string array
env = rlSimulinkEnv(mdl, agentBlocks, observationInfo, 
actionInfo); 

% Set reset function
env.ResetFcn = @(in)localResetFcn(in);

The error you encountered in the Reinforcement Learning Designer app—Items must be a 1-by-N cell array of character vectors or a string array—suggests that the input provided to the UI component is not in the expected format. To resolve this, ensure that when you add new agents, you are passing a string array or a cell array of character vectors. For example:

% Example of adding agents in the Reinforcement Learning Designer
agentNames = ["RL Agent1", "RL Agent2"]; % String array
% If using a cell array, it should be:
% agentNames = {'RL Agent1', 'RL Agent2'};

To enhance the robustness of your code, consider wrapping your environment creation in a try-catch block:

try
  env = rlSimulinkEnv(mdl, agentBlocks, observationInfo, 
  actionInfo);
catch ME
  disp('An error occurred while creating the environment:');
  disp(ME.message);
end

If further errors arise, please provide the specific error messages for additional troubleshooting assistance.

Feel free to reach out if you have any more questions or need further clarification on any of the steps outlined above.

[Muhammad]

Thank you very much for your suggestions; I will definitely review them and get back to you soon. In the meantime, I've written a new code for two agents in the `.slx` file, and their training is currently in progress.

[Umar]

Hi @Muhammad,

Thank you for your prompt response and for considering my suggestions. I appreciate your willingness to review them. I am pleased to hear that you have developed new code for the two agents in the .slx file and that their training is underway. I look forward to any updates you may have regarding their progress.

[Muhammad]

I tried to the above suggestion but still the problem persist with the RL designer app. Blk in above code was string and environment was successfully created but the problem with agent agent still persists.

[Umar]

Hi @Muhammad,

What are the names of the blocks you are using in Simulink model, and what problem are you encountering with agents, could you please provide some more details to help you resolve the problem.

[Muhammad]

Hey @Umar i mostly using matlab function blocks and RL_agent block with power electronic switches and load.

when adding an agent in App i got this error

Error in executing callback registered with ViewModel:

Error using matlab.ui.control.internal.model.AbstractStateComponent/set.Items (line 204)

'Items' must be a 1-by-N cell array of character vectors or a string array.

[Umar]

Hi @Muhammad,

This error typically arises when the Items property of a UI control (like a dropdown or list box) is not being set correctly. In MATLAB, the Items property must be defined as either a 1-by-N cell array of character vectors or as a string array. Here's how you can address this issue: Ensure that the data you're trying to assign to the Items property is formatted correctly. For example:

Cell Array

app.DropDown.Items = {'Item1', 'Item2', 'Item3'}; 

String Array

app.DropDown.Items = ["Item1", "Item2", "Item3"];

Verify the source of the data being assigned to Items. If it is being generated dynamically (e.g., from a simulation), print it out before assignment to confirm its structure. If you are using a function to generate this list, ensure that it returns the correct format. Here is an example of how to correctly set up a dropdown in your App:

function startupFcn(app) 
% Example items for the dropdown 
items = {'Agent1', 'Agent2', 'Agent3'}; 
app.DropDown.Items = items; % Assigning items correctly end

If you are setting Items within a callback function, make sure that it is being executed after all necessary variables are initialized and available.

Hope this helps.