Neural Network performance generalization

Hi to everybody! I´ve been trainning a neural network with above 9000 samples. My "divideParam" have been 50%/25%/25% for trainning, validation and test. After the trainning, regression plots show it is better than 0.999 for both of them. But, when I simulate my network with new input that the network "has not seen before",its performance fall down dramatically, becoming unacceptable. I wonder what´s the matter, and how can I check if it´s possible to improve the network or this kind of procedure (ANN), it´s not appropiate for this problem. Thanks for all!

Answer 1

[Greg Heath]

Either

your new data cannot be assumed to be a random sample from the same probability distribution as the trn/val/tst data

Or (much less likely)

you have too many hidden nodes and have memorized the training data.

Search Answers and Newsgroup for

Neq Nw

Hope this helps.

If it does, don't forget to officially accept this answer.

Greg

[Daniel]

Hi Greg, thanks. Since I´ve seen in Matlab Answers you´re an expert in NN, maybe I could specify about my problem so that you will be able to help me (firsty of all, I apolgize for my English, I´m Spanish).

1. I´ve above 9000 samples: 21-length input vector and 10-length output vector. When I train my network with this data block, results with unseen input are acceptable.

2. In fact, my physical problem must be well defined by 9 of theese 21 inputs, so I tried with them. I shocked when I check again my new nnetwork with the unseen inputs, and the MSE rose dramatically.

3. I´ve realized that problem maybe that all my 9000 patterns are included in three well defined bands and because of this validation and test performance are so good during the trainning but when I try with input that are not include in those bands exactly, results are so bad.

I´d like to know your opinion about whether this appreciation is wrong or not and also, if there is some relationship or estimation between the length of input and output vectors.

Thanks for all. Good work here in Matlab Answers