why is the neural network performance is not improving

I am using a feedforward neural network with 7 input vectors which are each matrices of 4 values and a target dataset of 4,600 values for one single output matrix. Keeping aside the output or simulation, my question is why is only my training performance reaching minimal value and not validation and testing performance, these two are remaining almost at a constant level.

[Walter Roberson]

If you were to over-train then the mse for the training data could drop a lot, but the test and validation might still have large errors.

I would wonder about whether the training data is full rank.

[Fazil Khan]

It is not my intention to overtrain the network actually, i want it to perform balanced in testing and validation as well. I divided the dataset in the ratio of 0.3; 0.35;0.35 respectively for that sake. My data set is a matrix of 4x1152 well scattered values, could you specify how do i make sure if it is full rank?

Answer 1

[Animesh Gupta]

Hello,

It is my understanding that validation and testing mean square error (MSE) are significantly higher compared to training MSE in this case.

When there is a significant difference between training and testing error, like this one, it is a case of over-fitting. There are many reasons for over-fitting. Some of them are:

• Not enough training data
• Complex model
• The training data is biased
• Continue training overshooting the generalization point (6th epoch in this case)

There are several methods to avoid over-fitting. One of them is early stopping. In this technique, when the validation error increases for a specified number of iterations, the training is stopped, and the weights and biases at the minimum of the validation error are returned.

Please refer the following documentation for more techniques to improve generalisation (training) and avoid overfitting - https://www.mathworks.com/help/deeplearning/ug/improve-neural-network-generalization-and-avoid-overfitting.html

I hope it helps.