Training a neural network

Hi,

I am trying to develop a neural network which predicts an output based on 4 inputs, one of which is the output of the previous step. Currently I am just using a standard function fitting network (not a time-series prediction).

The neural network works really well (r squared approx. 0.98 - 0.99) when the output of the previous step is given independent of the neural network result.

However, when I use the neural network predicted output as the input to the next prediction, the neural network result is virtually worthless. Also, the results differ greatly every time I re-train the network - i.e. it seems the results are very dependent on the initial weights.

I am not sure if this is a problem of overtraining? Any help would be greatly appreciated.

Sam

[Geoff]

Are you using neural net output as a predictor DURING training? Or do you train with the target value at T-1 and then synthesize that using the neural net once trained? I would opt for the latter.

[Sam harris]

Hi Geoff,

Thanks for your comment. I have been doing the latter, but still without much success. I then tried using a NARX network, however once I switch over to a closed loop prediction the results are still poor.

Is there a way to train the network using the neural net output as a predictor?

Thanks,

Sam

[Muhammad Qamar RAZA]

dear, any one knows how i can add or call particle swarm optimization pso training method or other training method In neural network toolbox

[Greg Heath]

Please post this as a new question.

[Greg Heath]

How many data points? How many hidden nodes? Is there a validation set for stopping? Do you get the same type of performace from a matlab demo data set?

Answer 1

[Greg Heath]

Sam harris on 2 Jul 2012

 % Create a Nonlinear Autoregressive Network with External Input
 % inputDelays = 1:1; feedbackDelays = 1:1; hiddenLayerSize = 10;

1. What makes you think these are appropriate inputs??

 % net =narxnet(inputDelays,feedbackDelays,hiddenLayerSize); 
 % net.inputs{1}.processFcns = {'removeconstantrows','mapminmax'};
 % net.inputs{2}.processFcns = {'removeconstantrows','mapminmax'};

2. Why bother? The last 2 statements are defaults.

 % [inputs,inputStates,layerStates,targets] = preparets(net,inputSeries,{},targetSeries); 
 % net.divideFcn = 'dividerand'; % Divide data randomly
 % net.divideMode = 'value'; % Divide up every value

3. The last 2 statements are inappropriate for time series

 % net.divideParam.trainRatio = 70/100; net.divideParam.valRatio = 15/100;
 % net.divideParam.testRatio = 15/100; 
 % net.trainFcn = 'trainlm'; % Levenberg-Marquardt 
 % net.performFcn = 'mse'; % Mean squared error 
 % net.plotFcns = {'plotperform','plottrainstate','plotresponse', ...
 % 'ploterrcorr', 'plotinerrcorr'};

4. Why bother? The last 6 statements are defaults.

 % % Train the Network 
 % [net,tr] =train(net,inputs,targets,inputStates,layerStates);
 % if true % code 
 % end

5. What does "if true ...etc... end" suppose to do?

6. You still have to close the loop and continue training.

7. See

http://www.mathworks.com/matlabcentral/newsreader/view_thread/332147#912806

Answer 2

[Greg Heath]

For the fitting net I assume you are using

 x =[input(:,2:end); target(:,1:end-1)];

 t = target(:,2:end);

size(input) = ?

size(target) = ?

numHidden = ?

net.divideParam = ?

R2trn ~ 0.985

R2val = ?

R2tst = ?

How is the timeseries net configured? Please include code.

Hope this helps.

Greg

[Sam harris]

Hi Greg,

Thanks for your help. I have changed to using a NARX network but am still having the same issue when I test the network as a closed loop. I thought perhaps I could try training the NN as a closed loop if this is possible (but I am not sure how). Attached is some of the code I have been using.

inputSeries = tonndata(Input,false,false);
targetSeries = tonndata(FOS,false,false);
Test1 = tonndata(Test1,false,false);
FOS1 = tonndata(FOS1,false,false);

% Create a Nonlinear Autoregressive Network with External Input inputDelays = 1:1; feedbackDelays = 1:1; hiddenLayerSize = 10; net = narxnet(inputDelays,feedbackDelays,hiddenLayerSize);

net.inputs{1}.processFcns = {'removeconstantrows','mapminmax'}; net.inputs{2}.processFcns = {'removeconstantrows','mapminmax'};

[inputs,inputStates,layerStates,targets] = preparets(net,inputSeries,{},targetSeries);

net.divideFcn = 'dividerand'; % Divide data randomly

net.divideMode = 'value'; % Divide up every value net.divideParam.trainRatio = 70/100; net.divideParam.valRatio = 15/100; net.divideParam.testRatio = 15/100;

net.trainFcn = 'trainlm'; % Levenberg-Marquardt

net.performFcn = 'mse'; % Mean squared error

net.plotFcns = {'plotperform','plottrainstate','plotresponse', ... 'ploterrcorr', 'plotinerrcorr'};

% Train the Network [net,tr] = train(net,inputs,targets,inputStates,layerStates);if true % code end

Many Thanks,

Sam

[Greg Heath]

You didn't answer my questions.

Greg

[Sam harris]

Hi Greg,

Thanks for your time, in answer to your questions:

size(input) = 12000 rows by 5 columns (data time series in rows)

numHidden = 10

net.divideParam = 70% used for training, 15% for validation, 15% for testing

R2trn ~ 0.985

R2val ~ 0.98

R2tst ~ 0.98

Code Used:

inputSeries = tonndata(Input,false,false);
targetSeries = tonndata(FOS,false,false);
Test1 = tonndata(Test1,false,false);
FOS1 = tonndata(FOS1,false,false);
inputDelays = 1:1;
feedbackDelays = 1:1;
hiddenLayerSize = 10;
net = narxnet(inputDelays,feedbackDelays,hiddenLayerSize);
eedback Pre/Post-Processing Functions
net.inputs{1}.processFcns = {'removeconstantrows','mapminmax'};
[inputs,inputStates,layerStates,targets] = preparets(net,inputSeries,{},targetSeries);
net.divideFcn = 'dividerand';
net.divideMode = 'value';
net.divideParam.trainRatio = 70/100;
net.divideParam.valRatio = 15/100;
net.divideParam.testRatio = 15/100;
net.trainFcn = 'trainlm';  % Levenberg-Marquardt
net.performFcn = 'mse';  % Mean squared error
net.plotFcns = {'plotperform','plottrainstate','plotresponse', ...
'ploterrcorr', 'plotinerrcorr'};
[net,tr] = train(net,inputs,targets,inputStates,layerStates);
outputs = net(inputs,inputStates,layerStates);
errors = gsubtract(targets,outputs);
performance = perform(net,targets,outputs)
trainTargets = gmultiply(targets,tr.trainMask);
valTargets = gmultiply(targets,tr.valMask);
testTargets = gmultiply(targets,tr.testMask);
trainPerformance = perform(net,trainTargets,outputs)
valPerformance = perform(net,valTargets,outputs)
testPerformance = perform(net,testTargets,outputs)
save('my_network','net'); %Saves the network
delay=1;
inputSeriesPred  = [inputSeries(end-delay+1:end),Test1];
targetSeriesPred = [targetSeries(end-delay+1:end),FOS1];
[Xs,Xi,Ai,Ts] = preparets(net,inputSeriesPred,{},targetSeriesPred);
yPred = net(Xs,Xi,Ai);
perf = perform(net,yPred,FOS1);
yPred=cell2mat(yPred);
ave=mean(FOSA);
for i=1:length(yPred);
  D(i,1)=(FOSA(i,1)-ave)^2;
  SSTOT=sum(D);
  D1(i,1)=(yPred(1,i)-FOSA(i,1))^2;
  SSERR=sum(D1);
end
RSOS=1-(SSERR/SSTOT);
RSOSif true
  % code
end

The network appears to be training fine until I use a closeloop network to test it.

Once again many thanks for your help,

Sam