Different training results for neural network when using full dataset versus partial dataset

I'm training a network using 'narxnet' and 'train'.

My training data is a part of a larger dataset. These are the two scenarios in which I get different results.

1. Trim the dataset so the entire input data is the training data. 'trainInd' = the entire dataset; no validation or test indices are provided
2. Use the entire dataset, but specify the training data by 'trainInd' (using the indices of the exact data from scenario 1); no validation or test indices are provided

The training terminates at the same conditions, and I'm using the same dataset, but I get different results. I've also experimented with adjusting the training data indices in scenario 2 based on # of delays specified with no luck.

Does anyone have any insight ino what might be causing this? (I'm aware with the issues of not specifying validation data, I'm just trying to replaicate behavior at the moment).

Answer 1

[Mrutyunjaya Hiremath]

The difference in results between scenario 1 and scenario 2 could be due to the different order of data samples seen during training. When you trim the dataset so that the entire input data is used for training (scenario 1), the network sees the data in the same order as it appears in the dataset. However, when you specify the training data using the indices (scenario 2), the network sees the data in a different order based on the selected indices.

In a neural network, the order in which data samples are presented during training can have an impact on the convergence and final performance of the model. Different orders of data samples can lead to different weight updates during training, potentially resulting in slightly different results.

To address this issue and ensure more consistent results, you can try the following:

1. Shuffle the dataset: Before creating the neural network and specifying the trainInd in scenario 2, shuffle the entire dataset randomly. This will help to randomize the order of data samples and potentially lead to more consistent training.
2. Set the random seed: If you are using a random number generator during training (e.g., weight initialization or mini-batch shuffling), set a fixed random seed before running both scenarios. This ensures that the randomization process during training is the same for both scenarios, leading to more reproducible results.

By shuffling the dataset and setting the random seed, you should get more consistent results between scenario 1 and scenario 2. Keep in mind that neural networks are still sensitive to other factors such as network architecture, learning rate, and training parameters, so it's possible to see slight differences even with these measures in place. However, the consistency should be improved.

[Katy]

Thank you for this answer! Really thorough and clear - makes sense.

Can I clarify:

1. Is there a shuffle function for 'train'? I see for 'trainNetwork' there are convient ways of shuffling with options to do this at every epoch (Training options for deep learning NN), but I haven't found an equivalent way of doing this with the 'train' function.
2. If there is not a way to do this within the 'train' function, there is the manual method of shuffling the data in the matrix. However, if I do that prior to specifiying 'trainInd', the data trained and validated doens't match the portions of the original data that I've intended after shuffling.

Or alterntaively, is there a way to use scenario 1, but provide the validation data as an entire datset as well as the training data as a separate dataset? I haven't found a way to do this using the options in 'train' function.

Hope this makese sense. Thanks!

[Mrutyunjaya Hiremath]

There is no direct built-in option to shuffle the training data when using the 'train' function for training neural networks. However, there are workarounds to achieve the desired shuffling behavior and provide separate training and validation datasets.

One approach to shuffle the data and provide separate training and validation datasets is to use the 'cvpartition' function to create custom cross-validation partitions. You can create custom partition objects for training and validation data and then use them with the 'train' function.

Here's an example of how to do it:

% Assuming you have your input data matrix 'X' and corresponding targets 'Y'

% Shuffle the data and targets
shuffledIndices = randperm(size(X, 1));
shuffledX = X(shuffledIndices, :);
shuffledY = Y(shuffledIndices, :);

% Create custom cross-validation partition for training and validation
cv = cvpartition(size(X, 1), 'HoldOut', 0.2); % 80% for training, 20% for validation

% Get training and validation indices from the partition
trainInd = training(cv);
valInd = test(cv);

% Split the shuffled data into training and validation datasets
trainingData = shuffledX(trainInd, :);
trainingTargets = shuffledY(trainInd, :);

validationData = shuffledX(valInd, :);
validationTargets = shuffledY(valInd, :);

% Train the neural network using the 'train' function
% Use the trainingData, trainingTargets for training
% Use the validationData, validationTargets for validation

[Katy]

Thank you for the clear explanation!

[Mrutyunjaya Hiremath]

You are most welcome :)