Reinforcement Learning : MaxSumWordLength is 65535 bits and a minimum word length of 65536 bits is necessary so that this sum or difference can be computed with no loss of precision - ( 'rl.simuli​nk.blocks.​AgentWrapp​er' )

Hello,

I am trying to use Reinforcement Learning to control the connection of the battery system of this example:

https://es.mathworks.com/help/physmod/sps/ug/simplified-model-of-a-small-scale-micro-grid.html

The RL Agent replaces the previous subsystem that generated the Control signal to the battery (this signal can be "1" (ON) for connecting the battery or "0" (OFF) for disconnecting it).

I was able to create a Integrated Environment from the Simulink model calling it "Integratedmicrogrid2_RL" and use it to train a DQN Agent following the instructions of the RL user guide without no many problems.

Then, I run my Simulink model using the RL Agent block as controller. It runs until it reaches a specific simulation time when an error happens (always at the same time). The Diagnostic Viewer shows the following error:

An error occurred while running the simulation and the simulation was terminated

Caused by:

• MATLAB System block 'Integratedmicrogrid2_RL/RL Agent/AgentWrapper' error occurred when invoking 'outputImpl' method of 'rl.simulink.blocks.AgentWrapper'. The error was thrown from ' 'C:\Program Files\MATLAB\R2020b\toolbox\rl\rl\+rl\+policy\AbstractPolicy.m' at line 237 'C:\Program Files\MATLAB\R2020b\toolbox\rl\rl\simulink\+rl\+simulink\+blocks\AgentWrapper.m' at line 98'.
• Invalid input argument type or size such as observation, reward, isdone or loggedSignals.
• MaxSumWordLength is 65535 bits and a minimum word length of 65536 bits is necessary so that this sum or difference can be computed with no loss of precision. Increase MaxSumWordLength if you want the sum to grow to a larger word length. To constrain the result to a type with a smaller word length, use colon-equals assignment, A(:) = B, or the CAST function.

I think that there must be a problem with the configuration of the memory during the simulation (maybe things related to buffer size?).

Any idea of parameters that I can modify to avoid this error?

Thanks in advance.

Answer 1

[Walter Roberson]

This is not a topic that I have worked on, so I might have the mechanics wrong.

The error is in growing the size of a fixed point object that has been configured for 65535 maximum bits, but it gets to a point where needs 65536 bits. If you were to increase the limit chances look like it would eventually need the new limit.

Why does it need to expand? This where I get to speculation.

It looks to me as if there is an internal calculation involving the sum or difference of two products, or a product and a constant or a sum. And that at least the product is configured at the fixed point level as an exact product. When you have two fixed point variables configured for exact multiplication, then bits(A*B) can be as large as bits(A)+bits(B). So for example if you had some 32 bit fixed point objects then a chain of N multiplications that size could require up to 32*N bits. And if you have two of these both growing, being multiplied by each other, then the number of bits can double each time, and after only 11 such multiplications, an original 32 bit number might need up to 65536 bits.

This happens when the fimath object associated with the variable does not specify approximation (and what to do on overflow or under flow). By default, fixed point objects do not have any floating point behaviour built in.

I do not know at the moment why fixed point is being used.

There are some systems of equations for which you can prove that a particular value will never exceed a particular range, such as through eigenvalue arguments or zpk analysis. In such cases it might be reasonable to code fixed point objects that discard trailing bits instead of trying to stay indefinitely accurate. But in other cases, you might need to configure a more floating point behavior.

The error message tells you that if you currently have C=A*B and you do not want C to extend automatically to hold all the implied bits, that C(:)=A*B is an idiom that signals truncation is acceptable.

[Juan Perez Torreglosa]

Thank you for your answer.

I am trying to find any block in my Simulink model that may extend its output to eliminate it. However, I think that the problem is related to the operation of the "agent". Maybe, anybody who knows how it works can tell us if it is extending its memory capacity with each step.

The error always happen in the 4466th time step.

[Juan Perez Torreglosa]

I modified the model changing the observations from 7 to 5 and made another training. I used the resulting agent to control the system and, again, the same error happened but in the 4880th time step:

An error occurred while running the simulation and the simulation was terminated

Caused by:

• MATLAB System block 'modelo_integrado3/RL Agent/AgentWrapper' error occurred when invoking 'outputImpl' method of 'rl.simulink.blocks.AgentWrapper'. The error was thrown from ' 'C:\Program Files\MATLAB\R2020b\toolbox\rl\rl\+rl\+policy\AbstractPolicy.m' at line 237 'C:\Program Files\MATLAB\R2020b\toolbox\rl\rl\simulink\+rl\+simulink\+blocks\AgentWrapper.m' at line 98'.
• Invalid input argument type or size such as observation, reward, isdone or loggedSignals.
• MaxSumWordLength is 65535 bits and a minimum word length of 65536 bits is necessary so that this sum or difference can be computed with no loss of precision. Increase MaxSumWordLength if you want the sum to grow to a larger word length. To constrain the result to a type with a smaller word length, use colon-equals assignment, A(:) = B, or the CAST function.

[Juan Perez Torreglosa]

I have tried to use a different agent. Now, after training a Policy Gradient (PG) Agent, I run the same model as before and now the error happens in the 2264th sample time.

The error is similar to previous ones:

An error occurred while running the simulation and the simulation was terminated

Caused by:

• MATLAB System block 'modelo_integrado3/RL Agent/AgentWrapper' error occurred when invoking 'outputImpl' method of 'rl.simulink.blocks.AgentWrapper'. The error was thrown from ' 'C:\Program Files\MATLAB\R2020b\toolbox\rl\rl\+rl\+policy\AbstractPolicy.m' at line 237 'C:\Program Files\MATLAB\R2020b\toolbox\rl\rl\simulink\+rl\+simulink\+blocks\AgentWrapper.m' at line 98'.
• Invalid input argument type or size such as observation, reward, isdone or loggedSignals.
• MaxSumWordLength is 65535 bits and a minimum word length of 65536 bits is necessary so that this sum or difference can be computed with no loss of precision. Increase MaxSumWordLength if you want the sum to grow to a larger word length. To constrain the result to a type with a smaller word length, use colon-equals assignment, A(:) = B, or the CAST function.

I though that maybe changing the type of agent I could avoid this memory consumption. Maybe the problem is in the model and not in the agent... But I don't think that a model provided by Matlab ( power_microgrid example) isn't debugged.