How to simulate default patternnet with feedforwardnet in Matlab?

I got very different training efficiency with the following network

    net = patternnet(hiddenLayerSize);

and the following one

    net = feedforwardnet(hiddenLayerSize, 'trainscg');
    net.layers{1}.transferFcn = 'tansig';
    net.layers{2}.transferFcn = 'softmax';
    net.performFcn = 'crossentropy';

on the same data.

I was thinking networks should be the same.

What thing I forgot?

UPDATE

The code below shows, that patternnet is systemtically outperforms feedforwardnet. This proves that feedforwardnet is initilized differently somehow. The question is what is the difference?

    hiddenLayerSize = 10;

    % pass 1, with patternnet
    net = patternnet(hiddenLayerSize);

    net.divideParam.trainRatio = 70/100;
    net.divideParam.valRatio = 15/100;
    net.divideParam.testRatio = 15/100;

    [net,tr] = train(net,x,t);

    y = net(x);
    performance = perform(net,t,y);

    fprintf('pass 1, patternnet, performance: %f\n', performance);
    fprintf('num_epochs: %d, stop: %s\n', tr.num_epochs, tr.stop);

    % pass 2, with feedforwardnet
    net = feedforwardnet(hiddenLayerSize, 'trainscg');
    net.layers{1}.transferFcn = 'tansig';
    net.layers{2}.transferFcn = 'softmax';
    net.performFcn = 'crossentropy';

    net.divideParam.trainRatio = 70/100;
    net.divideParam.valRatio = 15/100;
    net.divideParam.testRatio = 15/100;

    [net,tr] = train(net,x,t);

    y = net(x);
    performance = perform(net,t,y);

    fprintf('pass 2, feedforwardnet, performance: %f\n', performance);
    fprintf('num_epochs: %d, stop: %s\n', tr.num_epochs, tr.stop);

    % pass 1, with patternnet
    net = patternnet(hiddenLayerSize);

    net.divideParam.trainRatio = 70/100;
    net.divideParam.valRatio = 15/100;
    net.divideParam.testRatio = 15/100;

    [net,tr] = train(net,x,t);

    y = net(x);
    performance = perform(net,t,y);

    fprintf('pass 3, patternnet, performance: %f\n', performance);
    fprintf('num_epochs: %d, stop: %s\n', tr.num_epochs, tr.stop);

    % pass 2, with feedforwardnet
    net = feedforwardnet(hiddenLayerSize, 'trainscg');
    net.layers{1}.transferFcn = 'tansig';
    net.layers{2}.transferFcn = 'softmax';
    net.performFcn = 'crossentropy';

    net.divideParam.trainRatio = 70/100;
    net.divideParam.valRatio = 15/100;
    net.divideParam.testRatio = 15/100;

    [net,tr] = train(net,x,t);

    y = net(x);
    performance = perform(net,t,y);

    fprintf('pass 4, feedforwardnet, performance: %f\n', performance);
    fprintf('num_epochs: %d, stop: %s\n', tr.num_epochs, tr.stop);

Output follows:

    pass 1, patternnet, performance: 0.116445
    num_epochs: 353, stop: Validation stop.
    pass 2, feedforwardnet, performance: 0.693561
    num_epochs: 260, stop: Validation stop.
    pass 3, patternnet, performance: 0.116445
    num_epochs: 353, stop: Validation stop.
    pass 4, feedforwardnet, performance: 0.693561
    num_epochs: 260, stop: Validation stop.

[Andy Zhang]

Hi,Dear Olga Sorry to tell you i have no idea about your question. Recently,I am using function feedforwardnet to creat a Neural Network which is used to classify the facial expression ,but disappointed ,A question 'out of memory ,TYPE HELP MEMORY for your options'hapend when i train the Neural Network and the question is not exist when i use the function patternnet or newff. Do you have some ideas about my question? Thanks.

[Olga Lodnikova]

You probably should use less memory consuming train algorithms, like "trainscg" or "trainrp". Also probably you should downscale images and convert them to gray.

[Greg Heath]

If

 {I N ] = size(input)

 [O N ] = size(target)

and the network node topology is I-H-O, then if the number of training examples is

 Ntrn = (2/3)*N       % default is ~ 0.7*N

the number of training equations is

 Ntrneq = (2/3)*N*O

The number of unknown weights to estimate is

 Nw = (I+1)*H+(H+1)*O

Therefore, the number of equations is at least equal to the number of unknowns when

 Ntrn >= Ntrnlb = 1.5*Nw/O   % lb => lower bound

So, typically, robust designs result when

 Ntrn >~ 10*Ntrnlb ~  15*Nw/O

Consequently, keeping this in mind can prevent the use of more data than is nececessary.

Hope this helps.

Greg

Answer 1

[Greg Heath]

When comparing numerical designs, you should initialize the RNG to the same initial state before each training.

To compare the properties of FEEDFORWARDNET and PATTERNNET, compare the screen outputs of the following commands WITHOUT THE ENDING SEMICOLON

 net1 = feedforwardnet

 net2 = patternnet

 In particular, compare the following NINE fields

 transferFcn    = net.layers{2}.transferFcn
 yminparam      = net.outputs{2}.processParams{2}.ymin
 yminsetting    = net.outputs{2}.processSettings{2}.ymim
 yrangesetting  = net.outputs{2}.processSettings{2}.yrange
 performFcn     = net.performFcn
 plotFcns       = net.plotFcns
 plotParams     = net.plotParams
 trainFcn       = net.trainFcn
 trainParams    = net.trainParam

Therefore SIX additional fields besides the transferFcn, performFcn and trainFcn must be considered when trying to convert FEEDFORWARDNET to PATTERNNET and vice versa.

Hope this helps.

Thank you for formally accepting my answer

Greg

[Greg Heath]

I have verified Olga's performance results using her code on the iris_dataset with and without explicit RNG initialization (ERI) USING rng(0) before training.

WITHOUT ERI

 pass 1, patternnet, performance: 0.036678
num_epochs: 18, stop: Validation stop.
pass 2, feedforwardnet, performance: 0.491614
num_epochs: 15, stop: Validation stop.
pass 3, patternnet, performance: 0.036678
num_epochs: 18, stop: Validation stop.
pass 4, feedforwardnet, performance: 0.491614
num_epochs: 15, stop: Validation stop.

USING rng(0)

 pass 1, patternnet, performance: 0.033005
num_epochs: 17, stop: Validation stop.
pass 2, feedforwardnet, performance: 0.490139
num_epochs: 15, stop: Validation stop.
pass 3, patternnet, performance: 0.033005
num_epochs: 17, stop: Validation stop.
pass 4, feedforwardnet, performance: 0.490139
num_epochs: 15, stop: Validation stop.

Again the patternnet performance is better

NONETHELESS, I HAVE BEEN ABLE TO MODIFY OTHER PROPERTIES TO CONVERT FEEDFORWARDNET TO PATTERNNET AND VICE VERSA TO ACHEIVE THE EXACT SAME RESULTS.

HOWEVER, I CANNOT FIND THE FILE. IT TOOK QUITE A WHILE TO FIGURE IT OUT. SO IF I CAN'T FIND IT, IT WILL BE SOMETIME BEFORE I FEEL UP TO DUPLICATING THE WORK.

SORRY,

GREG

PS: If you type the commands WITHOUT THE ENDING SEMICOLON

 net = feedforwardnet

 net = patternnet

and compare the properties, you can eventually figure it out.