gpu limit on 3070 with a simple CNN

hello,

I have had this problem for the past two days and I have ran out of options how to solve this. I am training a basic CNN with the input and output mentioned in the code down below. However I use my data, wheter I split it up or use smaller samples, I always get the error " Out of memory on device. To view more detail about available memory on the GPU, use 'gpuDevice()'. If the problem persists, reset the GPU by calling 'gpuDevice(1)'." I also changed my minisizebatch to one, which doesn´t help neither. The last thing I tried is to run it on my CPU where it stops after the 3rd epoch out of three. Any Ideas?

imageInputsize = [6570 7000 1];
layers = [
         
           imageInputLayer(imageInputsize )
           convolution2dLayer(3,16, 'Padding', 'same' )
           batchNormalizationLayer
           reluLayer 
        
           convolution2dLayer(3,10, 'Padding', 'same' )
           batchNormalizationLayer
           reluLayer 
           
           resize3dLayer('OutputSize',[10 70000 5], 'Name', 'resize10x10x5')
           regressionLayer
];

options = trainingOptions('sgdm', ... 
    'InitialLearnRate', 0.01, ...
    'ExecutionEnvironment', 'gpu',...
    'MaxEpochs', 10,... 
    'MiniBatchSize', 1, ...
    'Shuffle', 'every-epoch',...
    'Verbose', false, ...
    'Plots', 'training-progress');

net = trainNetwork(arr_B,arr_V,layers, options);

note: the input as well the outputs are double arrays.

thank you so much

[Walter Roberson]

'resize10x10x5'

Your columns and panes do get scaled up by factors of 10 and 5... but your rows goes from 6570 to 10 total, which is downsizing by a factor of 657. That makes the name of the layer confusing to humans

[Kuno Bruswachtl]

ok, but that´s the problem I have with my input to output. i want to train it with a input of that size 6570x7000 where the 7000 are my samples. same goes for the output where the output then gets 70000 because of the multiplication with ten. However is this the problem with the gpu then?

[Walter Roberson]

As an experiment, try resize3dLayer('OutputSize',[10 7000 5]) and even resize3dLayer('OutputSize',[10 10 5])

You did not use 'Scale' so I would not expect that you are scaling up by a factor of 70000, but it would still be a good idea to test that out, to see if the resize is leading to problems.

[Kuno Bruswachtl]

thank you sir for your reply again.

I did try that out, but got the error 'the output size of the last layer doesn´t match the response size'. Just for reference as the feature input as well as the response I use an array. feature = 6570x7000 and response 10x 70000x5. So it cannot be that the this is too much data (I know it doesn´t make sense)?

[Kuno Bruswachtl]

update: when i reduce the filtersize and numFilters to 3,2. the whole program works. I just don´t know why :)

Answer 1

[Joss Knight]

You have 2-D data so you can't use a resize3dLayer. Use a resize2dLayer. But actually it doesn't look like that's what you want.

If you have 70000 samples, they will be the batch dimension not one of the spatial dimensions, so you shouldn't include that dimension at all when specifying your layers. You'll probably need to permute both the data and the response to S-by-C-by-B format (ie. permute the last two dimensions). Perhaps what you want is a series of 1-D convolutional layers?

It seems like your response requires your network to output a result of length 10 with 5 channels? So you're going to need to make your network reduce the spatial dimension to 10 and ensure that the last convolution has 5 filters. Use analyzeNetwork(layers) to check what you're doing and make it work.

[Kuno Bruswachtl]

first of all thank you for your answer. Although I have to say, I have now more questions than before.

to clarify I have 7000 samples of a 10x10x5 array, all saved in a cell array. I convert this cell array to an array because it doesn´t allow me to train with cells. Where the input is also a cell converted to an array.

Do still need to use 1D convolutions?

[Joss Knight]

Hi Kuno.

Unfortunately your questions indicate a level of unfamiliarity with Deep Learning that means it would be a heavy investment to get your code up to speed. I suggest you try some basic resources to get familiar with the principles:

• The MATLAB Deep Learning on-ramp
• The MATLAB Deep Learning e-book
• The MIT Deep Learning Book

From that point you should try the Deep Learning Examples from the MATLAB documentation, and also the documentation for trainNetwork specifically, focussing on the different kinds of input data it allows.

Failing this, if you have a commercial license you can contact MathWorks support to get help.

[Kuno Bruswachtl]

thank you so much for your kind response. I have done all that before.

My main problem is that there doesn´t seem to be a solution to use cells as a sequence input... do you have any suggestions to that? sorry this will be my last question...

[Joss Knight]

You say you have 10x10x5 data but your imageInputLayer takes an input size of 6570x7000x1. I'm not convinced you completely understand what your data is in terms of spatial, channel and batch dimensions. Similarly, to even be asking the question as to whether to be using 1-D, 2-D or 3-D convolutions implies that you're not sure what your data is or what the purpose of your network is. And to be attempting to match the network output to your responses by resizing the output data also indicates that you don't fully understand the purpose of the network, which is to transform the semantic information in the spatial input data into the space of the response (typically some metrics or categories), something which cannot be done just by warping the data.

Also, if you have read the documentation for trainNetwork then you will know all the different formats that are accepted, including this:

Numeric Array

For data that fits in memory and does not require additional processing like augmentation, you can specify a data set of images as a numeric array. If you specify images as a numeric array, then you must also specify the responses argument.

The size and shape of the numeric array depends on the type of image data.

DataFormat

2-D images

h-by-w-by-c-by-N numeric array, where h, w, and c are the height, width, and number of channels of the images, respectively, and N is the number of images.

3-D images

h-by-w-by-d-by-c-by-N numeric array, where h, w, d, and c are the height, width, depth, and number of channels of the images, respectively, and N is the number of images.

Perhaps if you want to use your data in cell form you might consider using an arrayDatastore:

ds = arrayDatastore(data,OutputType="same");

This begins to fit the formats for datastore inputs specified in the documentation. But you will have to ensure that your responses are included, perhaps using combine.

[Kuno Bruswachtl]

thank you kind sir for taking this time to reply to me. I will be looking in more carefully and figure it out. thanks a lot!