Mini-batch loss not changing for FCN

I have a Neural Network based on Fully Convolutional Neural Network for semantic segmentation ( FCN) and try to train it using Pascal-Context data. However, instead of 450 classes provided by the dataset, I only want to capture 12 + background.

I prepared a dataset with 7000+ images and label maps, and followed the tutorial on Semantic Segmentation ( Semantic Segmentation), and followed every step similarly to the one presented.

However, when I train my neural network my Mini-batch loss remains constant (checked until end of first 3 epochs, then tried to reduce the amount of images, but still the same value) and doesn't change, even if I change learning rate or other parameters. Is there something wrong with the training process?

So I tried to reduce the number of images to 400, and what I am getting is in picture below. It doesn't improve at all.

Below is my code:

% MAIN FUNCTION %

outputFolder=  fullfile('imade');
imgDir = fullfile('imade','images');
imds = imageDatastore(imgDir);

I = readimage(imds, 1);
% I = histeq(I);
% figure
% imshow(I)

labelDir = fullfile('imade','labels');
classes = [ 
    "bottle"
    "chair"
    "diningtable"
    "person"
    "pottedplant"
    "sofa"
    "tvmonitor" 
    "ground"
    "wall"
    "floor"
    "keyboard"
    "ceiling"
    "background"
    ];

valueSet = { 
    [230 25 75] 
    [60 180 75]
    [255 225 25]
    [0 130 200]
    [245 130 48]
    [145 30 180]
    [128 128 0]
    [210 245 60]
    [250 190 190]
    [0 128 128]
    [170 110 40]
    [128 0 0]
    [0 0 0]
  };
pxds = pixelLabelDatastore(labelDir,classes,valueSet);

% show sample image with overlay
C = readimage(pxds, 1);
cmap = camvidColorMap;
B = labeloverlay(I,C,'ColorMap',cmap);
figure
imshow(B)
pixelLabelColorbar(cmap,classes)

% calculate frequency of class pixels
tbl = countEachLabel(pxds);

% resize images and labels to desired format 
imageFolder = fullfile(outputFolder,'imagesReszed',filesep);
imds = resizeCamVidImages(imds,imageFolder);
labelFolder = fullfile(outputFolder,'labelsResized',filesep);
pxds = resizeCamVidPixelLabels(pxds,labelFolder);

% partition dataset into test and train
[imdsTrain, imdsTest, pxdsTrain, pxdsTest] = partitionCamVidData(imds,pxds);

% create FCN net
imageSize = [225 300];
numClasses = numel(classes);
lgraph = fcnLayers(imageSize,numClasses,'type','16s');
st = fullfile('imade','checkPoint');

% adjust based on occurrence of each class
imageFreq = tbl.PixelCount ./ tbl.ImagePixelCount;
classWeights = median(imageFreq) ./ imageFreq;
pxLayer = pixelClassificationLayer('Name','labels','ClassNames', tbl.Name, 'ClassWeights', classWeights);
lgraph = removeLayers(lgraph, 'pixelLabels');
lgraph = addLayers(lgraph, pxLayer);
lgraph = connectLayers(lgraph, 'softmax' ,'labels');

% set training parameters 
options = trainingOptions('sgdm', ...
    'Momentum', 0.9, ...
    'InitialLearnRate', 1e-3, ...
    'L2Regularization', 0.0005, ...
    'MaxEpochs', 100, ...  
    'MiniBatchSize', 1, ...
    'Shuffle', 'every-epoch', ...
    'VerboseFrequency', 2);

datasource = pixelLabelImageSource(imdsTrain,pxdsTrain);
doTraining = true;

if doTraining
    [net, info] = trainNetwork(datasource,lgraph,options);
else
    data = load (strcat(st,filesep,'convnet_checkpoint__4607__2018_01_27__21_25_03.mat'));
    war = data.net;
    [sp, info] = trainNetwork(datasource,war.Layers,options);
end

% Define the colormap used by CamVid dataset
function cmap = camvidColorMap()

cmap = [
    230 25 75 
    60 180 75
    255 225 25
    0 130 200
    245 130 48
    145 30 180
    128 128 0
    210 245 60
    250 190 190
    0 128 128
    170 110 40
    128 0 0
    0 0 0
    ];

% Normalize between [0 1].
cmap = cmap ./ 255;
end

% Add a colorbar to the current axis. The colorbar is formatted
% to display the class names with the color.

function pixelLabelColorbar(cmap, classNames)
colormap(gca,cmap)

% Add colorbar to current figure.
c = colorbar('peer', gca);

% Use class names for tick marks.
c.TickLabels = classNames;
numClasses = size(cmap,1);

% Center tick labels.
c.Ticks = 1/(numClasses*2):1/numClasses:1;

% Remove tick mark.
c.TickLength = 0;
end

% Partition CamVid data by randomly selecting 60% of the data for training. The
% rest is used for testing.

function [imdsTrain, imdsTest, pxdsTrain, pxdsTest] = partitionCamVidData(imds,pxds)

% Set initial random state for example reproducibility.
rng(0);
numFiles = numel(imds.Files);
shuffledIndices = randperm(numFiles);

% Use 60% of the images for training.
N = round(0.60 * numFiles);
trainingIdx = shuffledIndices(1:N);

% Use the rest for testing.
testIdx = shuffledIndices(N+1:end);

% Create image datastores for training and test.
trainingImages = imds.Files(trainingIdx);
testImages = imds.Files(testIdx);
imdsTrain = imageDatastore(trainingImages);
imdsTest = imageDatastore(testImages);

% Extract class and label IDs info.
classes = pxds.ClassNames;
labelIDs = 1:numel(pxds.ClassNames);

% Create pixel label datastores for training and test.
trainingLabels = pxds.Files(trainingIdx);
testLabels = pxds.Files(testIdx);
pxdsTrain = pixelLabelDatastore(trainingLabels, classes, labelIDs);
pxdsTest = pixelLabelDatastore(testLabels, classes, labelIDs);
end

% Resize images to desired image size.

function imds = resizeCamVidImages(imds, imageFolder)

if ~exist(imageFolder,'dir')
    mkdir(imageFolder)
else
    imds = imageDatastore(imageFolder);
    return; % Skip if images already resized
end

reset(imds)
while hasdata(imds)
    % Read an image.
    [I,info] = read(imds);

      % Resize image.
      I = imresize(I,[225 300]);

      % Write to disk.
      [~, filename, ext] = fileparts(info.Filename);
      imwrite(I,[imageFolder filename ext])
  end

imds = imageDatastore(imageFolder);
end

% Resize pixel label data to desired image size.

function pxds = resizeCamVidPixelLabels(pxds, labelFolder)

classes = pxds.ClassNames;
labelIDs = 1:numel(classes);
if ~exist(labelFolder,'dir')
    mkdir(labelFolder)
else
    pxds = pixelLabelDatastore(labelFolder,classes,labelIDs);
    return; % Skip if images already resized
end

reset(pxds)
while hasdata(pxds)
    % Read the pixel data.
    [C,info] = read(pxds);

      % Convert from categorical to uint8.
      L = uint8(C);

      % Resize the data. Use 'nearest' interpolation to
      % preserve label IDs.
      L = imresize(L,[225 300],'nearest');

      % Write the data to disk.
      [~, filename, ext] = fileparts(info.Filename);
      imwrite(L,[labelFolder filename ext])
  end

labelIDs = 1:numel(classes);
pxds = pixelLabelDatastore(labelFolder,classes,labelIDs);
end

Answer 1

[Birju Patel]

This is a R2017b bug in fcnLayers.

I've attached a patch for this issue. Here are the instructions for patching R2017b:

1. Download the attached patch, fcnpatch17b_2_23_2018.zip, and save it to your root MATLAB directory. The following command returns the root MATLAB directory.

% Root MATLAB directory
matlabroot

2. Execute the following MATLAB commands from the root MATLAB directory.

   unzip fcnpatch17b_2_23_2018.zip

3. Restart matlab and execute the following command:

rehash toolboxcache

NOTE: This bug has been fixed in the upcoming release, R2018a.

[Filip Kiniorski]

That is great testing it right now.

I also have a question regarding implementation. Original papers states that channels in the image should be shifted from RGB to BGR, is that performed in FCN layers, or does it need to be done separately?