MATLAB Answers

Index exceeds the number of array elements (1) When developing reinforcement learning environment

11 views (last 30 days)
Abubakar Ali
Abubakar Ali on 27 Oct 2020
Commented: Abubakar Ali on 30 Oct 2020
Below is the main program of my code. I am trying to implement reinforcement learning environment using function handles. The error is in line 44 where the environment is to be validated. I think the error is due to the fact that my observation is a 4by1 cell not numeric.
clc
clear
%%
% Initialize Observation settings
ObservationInfo = rlNumericSpec([4 1]);
ObservationInfo.Name = 'UAV States';
ObservationInfo.Description = 'T, Mu, UAV_Tour';
% Initialize Action settings
ActionInfo = rlFiniteSetSpec(1:2^7);
ActionInfo.Name = 'UAV Action';
%% Defining Environment Constants
N = 10;
envConstants = Initlize_parameters(N);
[Devices_Locations,C,W,L_j,R,R_ik,ell_ik,ell_i0,F,Dockin_Location]=System_Model_Setup(N,envConstants);
envConstants.Devices_Locations = Devices_Locations;
envConstants.C = C;
envConstants.W = W;
envConstants.L_j =L_j;
envConstants.R = R;
envConstants.R_ik = R_ik;
envConstants.ell_ik = ell_ik;
envConstants.ell_i0 = ell_i0;
envConstants.F = F;
envConstants.Dockin_Location = Dockin_Location;
% Reward each time step the energy is better
envConstants.RewardForBetter = 1;
% Reward each time step the energy is the same
envConstants.RewardForSame = 0;
% Penalty when the energy is worse
envConstants.PenaltyForWorse = -1;
% Penalty when the energy is infinity
envConstants.PenaltyForInfeasible = -10;
bestE = inf;
%%
[InitialObservation,LoggedSignals] = uavResetFunction(N,envConstants);
[Observation,Reward,IsDone,LoggedSignals] = uavStepFunction(100,LoggedSignals,N,envConstants);
if bestE>Observation{4}
bestE = Observation{4}
end
ResetHandle = @()uavResetFunction(N,envConstants);
StepHandle = @(Action,LoggedSignals) uavStepFunction(Action,LoggedSignals,N,envConstants);
%%
env = rlFunctionEnv(ObservationInfo,ActionInfo,StepHandle,ResetHandle);
%% Validating environment
InitialObs = reset(env)
[NextObs,Reward,IsDone,LoggedSignals] = step(env,100);
NextObs
%% RESET FUNCTION
function [InitialObservation, LoggedSignal] = uavResetFunction(N,envConstants)
% Reset function to place custom cart-pole environment into a random
% initial state.
%Devices classification
T0 = [ones(N,1) zeros(N,1)];
% Devices association (No association)
Mu0 = zeros(N);
% UAV_Tour (To all devices)
UAV_Tour0 = [envConstants.Dockin_Location;envConstants.Devices_Locations;envConstants.Dockin_Location];
% Initial energy
E_DQN = inf;
% Return initial environment state variables as logged signals.
LoggedSignal.State = {T0;Mu0;UAV_Tour0;E_DQN};
InitialObservation = LoggedSignal.State;
end
%% STEP FUNCTION
function [NextObs,Reward,IsDone,LoggedSignals] = uavStepFunction(Action,LoggedSignals,N,envConstants)
% Custom step function to construct cart-pole environment for the function
% handle case.
%
% This function applies the given action to the environment and evaluates
% the system dynamics for one simulation step.
% Check if the given action is valid.
% MinT = 1;
% MaxT = 2^(N-1);
% if ~ismember(Action,[MinT MaxT])
% error('Action must be between %g and %g.', MinT,MaxT);
% end
Tdec = Action;
% Unpack the state cell from the logged signals.
State = LoggedSignals.State;
E_best = State{4};
[T,Mu,UAV_Tour,E_DQN] = DQN_Solution(Tdec,N,envConstants);
% Check terminal condition.
IsDone = E_DQN == inf;
% Get reward.
if IsDone
Reward = envConstants.PenaltyForInfeasible;
elseif (E_best > E_DQN)
Reward = envConstants.RewardForBetter;
E_best = E_DQN;
elseif (E_best == E_DQN)
Reward = envConstants.RewardForSame;
else
Reward = envConstants.RewardForWorse;
end
% Making the new state observations
State{1} = T;
State{2} = Mu;
State{3} = UAV_Tour;
State{4} = E_best;
LoggedSignals.State = State;
% Transform state to observation.
NextObs = LoggedSignals.State;
end

  2 Comments

KSSV
KSSV on 27 Oct 2020
This error occurs when you try to extract more number of elements than present in an array.
Example:
A = rand(1,10) ;
A(1)
A(7)
A(10)
A(11) % error, becuase there are only 10 elements in A
Check your data dimensions and use the indexing.
Abubakar Ali
Abubakar Ali on 27 Oct 2020
I get your point, but I don't think it's the case here. the issue I think has to do with the type of data rlFunctionEnv is expecting. In the documentation they used rlNumeric with double data type as the observationInfo, in my case, the observationInfo is a cell.

Sign in to comment.

Answers (1)

Cris LaPierre
Cris LaPierre on 27 Oct 2020
The error suggests that ct>1, but either/both obj and Data have a length of 1. This is what the (1) at the end of the 'Index exceeds the number of array elements' error message is indicating.
The code you shared does not help debug this because the necessary code is buried in your class.

  1 Comment

Abubakar Ali
Abubakar Ali on 30 Oct 2020
I get your point. The issue is with rlNumericspec matlab built in function. My observation space is a cell not a single numeric value. I read the documentation and I have seen an implementation of simplependulumwithimage-discrete which also has an image as an observation. I tried to look into the code of the simplependulwithimages environment but I couldn't.

Sign in to comment.

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by