Difference (mistake) between two images

Hello, I need help with a Master project. I need rectangle human mistake, in connector seals.

In picture 1.jpg, I have a connector with 17 white seals, this is model1.

In picture 2.jpg, I have a connector with 16 white seals, this is model2. (MODEL WITH MISTAKE, 1 missing white seal)

In autocad drawing i want to color or choose what model will be, somehow automatic combine (register) it with the model2 or new image with mistake, and rectangle that place, like in the last image. All images will be created from same height and angle.

so long i can to bwlabel the drawing(image) and choose what circles i want to paint.

rectangled mistake.

I cant use imregcorr to register autocad drawing with model1 because, its working just on rgb image, and i cant bwlabel the autocaddrawing without imbinarizing it. So thx for your time and waiting for answer.

Answer 1

[DGM]

This isn't exactly production ready, but it's a rough example. This uses CVT tools to do the registration estimation. I don't know how robust this will be with the given settings in the registration function. I don't have CVT and I don't have but the one image to try it on.

% get the images, discard color
good = imread('good.jpg');
bad = imread('mistake.jpg');
good = rgb2gray(good);
bad = rgb2gray(bad);

% detect the plugs
goodplug = imfill(imbinarize(good),'holes');
badplug = imfill(imbinarize(bad),'holes');
goodplug = bwareaopen(goodplug,100);
badplug = bwareaopen(badplug,100);

% remove the plugs from both images
goodnoplug = good;
goodnoplug(goodplug) = 0;
badnoplug = bad;
badnoplug(badplug) = 0;

montage({goodnoplug badnoplug})

% attempt to register the images (requires CVT)
movreg = registerImages_connector(badnoplug,goodnoplug);

% use the image registration to register the plug masks
badplugadj = imwarp(badplug,movreg.InitialRefObj,movreg.Transformation, ...
            'OutputView',movreg.ResultRefObj,'SmoothEdges', true);

montage({goodplug badplugadj})

% split the blobs in the good plug mask
goodplug = bwdist(~goodplug)>=5; % adjust radius if necessary

imshow(goodplug)

% find blob centroids as linear indices
S = regionprops(goodplug,'centroid');
plugidx = round(vertcat(S.Centroid));
plugidx = sub2ind(size(goodplug),plugidx(:,2),plugidx(:,1));

% check those locations in the registered image under test
plugismissing = ~badplugadj(plugidx)

plugismissing = 17×1 logical array
   0
   0
   0
   1
   0
   0
   0
   0
   0
   0
   0
   0
   0
   0
   0
   0
   0

[DGM]

Just for sake of completeness, I decided to do this without the registration tools. Note that in this example, I'm preprocessing the reference image in whole before doing anything with the image under test. That's really the way the above example should be organized, but I think for the sake of demonstration processing the images together makes it easier to see that they're being processed in the same way.

In this example, I'm going to use a modified working image. I've added extra rotation and added an extra plug where it doesn't belong. That way we can check for both missing and extra plugs.

% %%%% PROCESS REFERENCE IMAGE %%%%

% get the image, discard color
reference = imread('good.jpg');
reference = rgb2gray(reference);

% detect the plugs
refplug = imfill(imbinarize(reference),'holes');
refplug = bwareaopen(refplug,100);

% remove the plugs from image
refnoplug = reference;
refnoplug(refplug) = 0;

% detect holes
holeth = 20;
maxspecklearea = 100;
solidityrange = [0.7 1];
eccentrange = [0 0.95];
refholes = refnoplug <= holeth;
refholes = ~bwareaopen(~refholes,maxspecklearea);
refholes = bwareaopen(refholes,maxspecklearea);
refholes = bwpropfilt(refholes,'solidity',solidityrange);
refholes = bwpropfilt(refholes,'eccentricity',eccentrange);
imshow(refholes)

% calculate convex hull and object rotation
refholes = bwconvhull(refholes);
S = regionprops(refholes,'minferetproperties');
thref = S(1).MinFeretAngle - 90;

% correct images for rotation
refholes = imrotate(refholes,thref);
refplug = imrotate(refplug,thref);
montage({refholes refplug})

% close-crop plug mask based on extent of convhull
S = regionprops(refholes,'boundingbox');
rect = round(S.BoundingBox);
xrange = rect(1):sum(rect([1 3]));
yrange = rect(2):sum(rect([2 4]));
refplug = refplug(yrange,xrange);

% get template (this is a label array)
template = imread('holetemplate.png');
szref = [size(template,1) size(template,2)];

% find hole centers as linear indices into image
S = regionprops(template,'centroid');
holeidx = round(vertcat(S.Centroid));
holeidx = sub2ind(szref,holeidx(:,2),holeidx(:,1));

% resize reference image to match template
refplug = imresize(refplug,szref);

% check hole locations in the reference image
% this tells us which plugs to look for in the working image
refplugidx = refplug(holeidx);


% %%%% PROCESS WORKING IMAGE %%%%

% get the image, discard color
testimg = imread('rotatedbad.jpg');
testimg = rgb2gray(testimg);

% detect the plugs
testplug = imfill(imbinarize(testimg),'holes');
testplug = bwareaopen(testplug,100);

% remove the plugs from image
testnoplug = testimg;
testnoplug(testplug) = 0;

% detect holes
holeth = 20;
maxspecklearea = 100;
solidityrange = [0.7 1];
eccentrange = [0 0.95];
testholes = testnoplug <= holeth;
testholes = ~bwareaopen(~testholes,maxspecklearea);
testholes = bwareaopen(testholes,maxspecklearea);
testholes = bwpropfilt(testholes,'solidity',solidityrange);
testholes = bwpropfilt(testholes,'eccentricity',eccentrange);
imshow(testholes)

% calculate convex hull and object rotation
testholes = bwconvhull(testholes);
S = regionprops(testholes,'minferetproperties');
thtest = S(1).MinFeretAngle - 90;

% correct images for rotation
testholes = imrotate(testholes,thtest);
testplug = imrotate(testplug,thtest);
montage({testholes testplug})

% close-crop plug mask based on extent of convhull
S = regionprops(testholes,'boundingbox');
rect = round(S.BoundingBox);
xrange = rect(1):sum(rect([1 3]));
yrange = rect(2):sum(rect([2 4]));
testplug = testplug(yrange,xrange);

% resize moving image to match reference
testplug = imresize(testplug,szref);
montage({refplug testplug})

% check hole locations in the working image
testplugidx = testplug(holeidx);

% these are linear indices into the template where a plug is missing
plugismissing0 = refplugidx & ~testplugidx; % index into arrays returned by regionprops()
plugismissing = template(holeidx(plugismissing0)) % index into hole template

plugismissing = uint8 18

% these are linear indices into the template where a plug is misplaced
plugismisplaced0 = ~refplugidx & testplugidx; % index into arrays returned by regionprops()
plugismisplaced = template(holeidx(plugismisplaced0)) % index into hole template

plugismisplaced = uint8 21

% plot the location of missing plugs
imshow(testplug); hold on
[y x] = ind2sub(szref,holeidx(plugismissing0));
plot(x,y,'o','linewidth',3,'markersize',10)
[y x] = ind2sub(szref,holeidx(plugismisplaced0));
plot(x,y,'x','linewidth',3,'markersize',10)

In this example, the template is a synthetic image, so regionprops(), etc will order the blobs in a predictable way. Though it's unnecessary here, the way plugismissing is calculated ignores the blob ordering from anything returned by regionprops() and gets the index from the template itself.

Indices in the template are ordered columnwise like array indices. Hole #18 would be the second row, fifth column.

[Image Analyst]

Since the 1 in that vector depends on the labeling, and the labeling depends on what spots are there or missing, I think @Audrius Bieliunas would like to know how to turn plugismissing into a (row, column) coordinate, like (for this example) row 2, column 5 in the spot array is the location that is different (spot where there should be no spot, or no spot where there should be a spot).

[DGM]

I figured that since the label-template mapping is all handled on the reference that it'd be easy enough for OP to figure out -- especially considering that I don't really know how he wants to manage the template or mapping. The mapping doesn't even have to be automated.

That said, I suppose this is still the most interesting thread at the moment. I might as well keep going.

EDIT: I improved the mess. I simply manually created a template image to which both images are registered. The template also serves as a label array. This way, all holes can be tested, but it's only the difference in plug presence between the test and reference images that's important. Label order is defined by the template, so it won't change.

[Image Analyst]

Yes the best way is to use a template. I'd like to assume that the user has some kind of control over the image capture conditions to ensure consistent lighting and positioning. For example some kind of a jig, like a L-shaped bracket that the user can shove the part into, and a camera rigidly secured to some kind of framework. That way the part is in the same position in the image every time and you can simply use imabsdiff() to find the difference between the current piece and the reference piece. Then the centroid of the difference blob can be used directly to determine what row and column of the piece it is in.

[Audrius Bieliunas]

I am really appreciate for the help with code and all the instructions, you both guys are really amazing. One more question, is there way, for not use the reference = imread('good.jpg');like in the second code, but use some coordinates or array that i could choose what blob i want to be white, and dont picture the reference image every time the model had changed. I mean not find difference by good.jpg and bad.jpg, but maybe find difference from labeled template and bad.jpg, or maybe from full black hole image to bad.jpg. Something where i can change the model, and dont need to retake reference picture to find the difference, like in below image, thx for your answer.

[Audrius Bieliunas]

and DGM, can u please sen the code, how did you make that template in the second code? Thank you for your answer.

[DGM]

You'll still need a reference image for registering, but you don't need to get the plug positions from the reference.

In this example, any plugs in the reference image are ignored. The plug locations are specified in an index vector.

% %%%% PROCESS REFERENCE IMAGE %%%%

% get the image, discard color
reference = imread('good.jpg');
reference = rgb2gray(reference);

% detect the plugs
refplug = imfill(imbinarize(reference),'holes');
refplug = bwareaopen(refplug,100);

% remove the plugs from image
refnoplug = reference;
refnoplug(refplug) = 0;

% detect holes
holeth = 20;
maxspecklearea = 100;
solidityrange = [0.7 1];
eccentrange = [0 0.95];
refholes = refnoplug <= holeth;
refholes = ~bwareaopen(~refholes,maxspecklearea);
refholes = bwareaopen(refholes,maxspecklearea);
refholes = bwpropfilt(refholes,'solidity',solidityrange);
refholes = bwpropfilt(refholes,'eccentricity',eccentrange);

% calculate convex hull and object rotation
refhull = bwconvhull(refholes);
S = regionprops(refhull,'minferetproperties');
thref = S(1).MinFeretAngle - 90;

% correct images for rotation
refhull = imrotate(refhull,thref);
refplug = imrotate(refplug,thref);

% close-crop plug mask based on extent of convhull
S = regionprops(refhull,'boundingbox');
rect = round(S.BoundingBox);
xrange = rect(1):sum(rect([1 3]));
yrange = rect(2):sum(rect([2 4]));
refplug = refplug(yrange,xrange);

% stuff needed to create the template
%refholes = imrotate(refholes,thref); % <-- need to carry this to make the template
%refholes = refholes(yrange,xrange); % <-- this is used to manually create the template
%imwrite(refholes,'refholes.png') % <-- write it and open it externally

% get template (this is a label array)
template = imread('holetemplate.png');
szref = [size(template,1) size(template,2)];

% find hole centers as linear indices into image
S = regionprops(template,'centroid');
holeidx = round(vertcat(S.Centroid));
holeidx = sub2ind(szref,holeidx(:,2),holeidx(:,1));

% explicitly specify which holes should be filled
% note that i specified 5 and 12 instead of 6 and 11
pluggedholes = [5 12 13 18 19 24 27 29 31 34 35 42 43 45 51 53 56];

% generate logical map of plugs from index list
refplugidx = false(size(holeidx));
refplugidx(pluggedholes) = true;



% %%%% PROCESS WORKING IMAGE %%%%

% get the image, discard color
testimg = imread('rotatedbad.jpg');
testimg = rgb2gray(testimg);

% detect the plugs
testplug = imfill(imbinarize(testimg),'holes');
testplug = bwareaopen(testplug,100);

% remove the plugs from image
testnoplug = testimg;
testnoplug(testplug) = 0;

% detect holes
holeth = 20;
maxspecklearea = 100;
solidityrange = [0.7 1];
eccentrange = [0 0.95];
testholes = testnoplug <= holeth;
testholes = ~bwareaopen(~testholes,maxspecklearea);
testholes = bwareaopen(testholes,maxspecklearea);
testholes = bwpropfilt(testholes,'solidity',solidityrange);
testholes = bwpropfilt(testholes,'eccentricity',eccentrange);

% calculate convex hull and object rotation
testhull = bwconvhull(testholes);
S = regionprops(testhull,'minferetproperties');
thtest = S(1).MinFeretAngle - 90;

% correct images for rotation
testhull = imrotate(testhull,thtest);
testplug = imrotate(testplug,thtest);

% close-crop plug mask based on extent of convhull
S = regionprops(testhull,'boundingbox');
rect = round(S.BoundingBox);
xrange = rect(1):sum(rect([1 3]));
yrange = rect(2):sum(rect([2 4]));
testplug = testplug(yrange,xrange);

% resize moving image to match reference
testplug = imresize(testplug,szref);

% check hole locations in the working image
testplugidx = testplug(holeidx);

% these are linear indices into the template where a plug is missing
plugismissing0 = refplugidx & ~testplugidx; % index into arrays returned by regionprops()
plugismissing = template(holeidx(plugismissing0)) % index into hole template

% these are linear indices into the template where a plug is misplaced
plugismisplaced0 = ~refplugidx & testplugidx; % index into arrays returned by regionprops()
plugismisplaced = template(holeidx(plugismisplaced0)) % index into hole template

% plot the location of missing plugs
imshow(testplug); hold on
[y x] = ind2sub(szref,holeidx(plugismissing0));
plot(x,y,'o','linewidth',3,'markersize',10)
[y x] = ind2sub(szref,holeidx(plugismisplaced0));
plot(x,y,'x','linewidth',3,'markersize',10)

As to how I created the template/label array, I needed to carry a copy of the refholes image through all the geometric transformations without taking the convex hull of it. It was something that I did in an ad-hoc fashion at the command line, but I rewrote/renamed things to include it in the above code. In this code, there are three commented lines which are only used to generate the image used for generating the template.

At that point, I have this image on disk:

I opened it in GIMP and created two overlying black layers. Adjust opacity so that you can see what you're doing. In one layer, I drew horizontal lines. In the other layer, I drew vertical lines. I set the upper layer to 'multiply', giving me the intersections of the lines.

It's important the vertical bars are perfectly vertical. That's what allows the labeling step to work easily. Otherwise you'd need to do extra things to get the labels ordered in a way that made sense. Save the image as a grayscale PNG. No JPG. Never JPG. FWIW, I attached the XCF file, but I doubt you'd need it for anything.

This could be done in Inkscape or anything that can draw a line or rectangle with orthogonal constraint or has any other way of creating vertically-aligned dots (their left edges needs to be aligned for the labeler to order them in sequence).

To convert this template to a label image, I just did:

% read the binarized template exported from GIMP
A = imread('holetemplate_nolabel.png');

% label it and convert it to integer class
% this will accept up to 255 blobs
L = uint8(bwlabel(A));

% save it
% this is now our labeled template image
imwrite(L,'holetemplate.png')

... and that's it.

Alternatively, I could have relied on the refhole image and gone off of its blob centroids to create a dot template, but it wouldn't have been very regular, and it would have needed extra steps to order the labels.

[Image Analyst]

@Audrius Bieliunas that labeling is kind of messed up because of the way labeling happens. Since the blobs are tilted, blob 2 shows up at the bottom left. See attached demo.

You need to use kmeans to figure out what row and column of your template the blobs is and relabel them appropriately.

[DGM]

Note that IA is referring to your image with the yellow markings on it. That lack of control over natural label ordering is why I opted to create a synthetic image.

FWIW, this is how the ordering goes with the example template:

[Audrius Bieliunas]

Thank you really much, you are my hero.

[DGM]

If and when you feel this satisfies your question, you can accept the answer. Accepting an answer may help the next person choose to read this thread. If you have further questions or wish to wait before making that decision, that's fine too.

FWIW, I'd have to send some thanks back to you too. This was kind of a good question to work with.

[DGM]

... story of my life.

[Audrius Bieliunas]

DGM u sir save the day. Would u suggest something how to send these markers form testplug to testimg? I understand that i have to align both images?

[DGM]

Are you saying to put the X and O markers on the photograph instead of the mask?

That could be done if that's what you're after

% %%%% PROCESS REFERENCE IMAGE %%%%

% get the image, discard color
reference = imread('good.jpg');
reference = rgb2gray(reference);

% detect the plugs
refplug = imfill(imbinarize(reference),'holes');
refplug = bwareaopen(refplug,100);

% remove the plugs from image
refnoplug = reference;
refnoplug(refplug) = 0;

% detect holes
holeth = 20;
maxspecklearea = 100;
solidityrange = [0.7 1];
eccentrange = [0 0.95];
refholes = refnoplug <= holeth;
refholes = ~bwareaopen(~refholes,maxspecklearea);
refholes = bwareaopen(refholes,maxspecklearea);
refholes = bwpropfilt(refholes,'solidity',solidityrange);
refholes = bwpropfilt(refholes,'eccentricity',eccentrange);

% calculate convex hull and object rotation
refhull = bwconvhull(refholes);
S = regionprops(refhull,'minferetproperties');
thref = S(1).MinFeretAngle - 90;

% correct images for rotation
refhull = imrotate(refhull,thref);
refplug = imrotate(refplug,thref);

% close-crop plug mask based on extent of convhull
S = regionprops(refhull,'boundingbox');
rect = round(S.BoundingBox);
xrange = rect(1):sum(rect([1 3]));
yrange = rect(2):sum(rect([2 4]));
refplug = refplug(yrange,xrange);

% stuff needed to create the template
%refholes = imrotate(refholes,thref); % <-- need to carry this to make the template
%refholes = refholes(yrange,xrange); % <-- this is used to manually create the template
%imwrite(refholes,'refholes.png') % <-- write it and open it externally

% get template (this is a label array)
template = imread('holetemplate.png');
szref = [size(template,1) size(template,2)];

% find hole centers as linear indices into image
S = regionprops(template,'centroid');
holeidx = round(vertcat(S.Centroid));
holeidx = sub2ind(szref,holeidx(:,2),holeidx(:,1));

% explicitly specify which holes should be filled
% note that i specified 5 and 12 instead of 6 and 11
pluggedholes = [5 12 13 18 19 24 27 29 31 34 35 42 43 45 51 53 56];

% generate logical map of plugs from index list
refplugidx = false(size(holeidx));
refplugidx(pluggedholes) = true;



% %%%% PROCESS WORKING IMAGE %%%%

% get the image, discard color
testimg = imread('rotatedbad.jpg');
testimg = rgb2gray(testimg);

% detect the plugs
testplug = imfill(imbinarize(testimg),'holes');
testplug = bwareaopen(testplug,100);

% remove the plugs from image
testnoplug = testimg;
testnoplug(testplug) = 0;

% detect holes
holeth = 20;
maxspecklearea = 100;
solidityrange = [0.7 1];
eccentrange = [0 0.95];
testholes = testnoplug <= holeth;
testholes = ~bwareaopen(~testholes,maxspecklearea);
testholes = bwareaopen(testholes,maxspecklearea);
testholes = bwpropfilt(testholes,'solidity',solidityrange);
testholes = bwpropfilt(testholes,'eccentricity',eccentrange);

% calculate convex hull and object rotation
testhull = bwconvhull(testholes);
S = regionprops(testhull,'minferetproperties');
thtest = S(1).MinFeretAngle - 90;

% correct images for rotation
testhull = imrotate(testhull,thtest);
testplug = imrotate(testplug,thtest);
testimg = imrotate(testimg,thtest);

% close-crop plug mask based on extent of convhull
S = regionprops(testhull,'boundingbox');
rect = round(S.BoundingBox);
xrange = rect(1):sum(rect([1 3]));
yrange = rect(2):sum(rect([2 4]));
testplug = testplug(yrange,xrange);
testimg = testimg(yrange,xrange);

% resize moving image to match reference
testplug = imresize(testplug,szref);
testimg = imresize(testimg,szref);

% check hole locations in the working image
testplugidx = testplug(holeidx);

% these are linear indices into the template where a plug is missing
plugismissing0 = refplugidx & ~testplugidx; % index into arrays returned by regionprops()
plugismissing = template(holeidx(plugismissing0)) % index into hole template

% these are linear indices into the template where a plug is misplaced
plugismisplaced0 = ~refplugidx & testplugidx; % index into arrays returned by regionprops()
plugismisplaced = template(holeidx(plugismisplaced0)) % index into hole template

% plot the location of missing plugs
imshow(testimg); hold on
[y x] = ind2sub(szref,holeidx(plugismissing0));
plot(x,y,'o','linewidth',3,'markersize',10)
[y x] = ind2sub(szref,holeidx(plugismisplaced0));
plot(x,y,'x','linewidth',3,'markersize',10)

[Audrius Bieliunas]

thank you one more time.

[Audrius Bieliunas]

Hello misters, after month then you help me, I am still little struggling to imbinarize transperency blobs, because they are the same color as background. I tried yours Image Analyst DeltaE color segmentation, but it seems i cant do it right. I just want to imbinarize those transperency blobs that later to insert in code that sir @DGM gave me. Could you quide me?. And if ewrything is ok in here, how to imwrite the image?

rgbImage = imread('geras.jpg');
[rows, columns, numberOfColorChannels] = size(rgbImage);
numberOfPixels = rows*columns;

[mask, maskedRGBImage] = createMask(rgbImage);
% Take largest blob only.
mask = bwareafilt(mask, 1);
% Fill any possible holes.
mask = imfill(mask, 'holes');
% Get masked image again with new mask
% Mask image by multiplying each channel by the mask.
maskedRgbImage = rgbImage .* cast(mask, 'like', rgbImage);

lesionBoundary = bwboundaries(mask);

lab_Image = rgb2lab(rgbImage);
% Extract out the color bands from the original image
% into 3 separate 2D arrays, one for each color component.
[lImage, aImage, bImage] = imsplit(lab_Image);

[LMean, aMean, bMean] = GetMeanLABValues(lImage, aImage, bImage, mask);

darkbrown = uint8([101 67 33]);

refLab_db = rgb2lab(darkbrown);

%  Create the Delta E image for dark brown.
% This is an image that represents the color difference.
deImage_db = GetDeltaEImage(lImage, aImage, bImage, refLab_db);
figure
imshow(deImage_db, [  ]);

Image i got:

[DGM]

I'm not sure what part of the process this is. Are you trying to be able to pick up the three brown large plugs so that they can be processed like the white plugs? I kind of thought those were just blank cavities.

If so, are the plugs in the large holes always brown, or are there white plugs for those holes as well?

[Audrius Bieliunas]

I am trying to filter those transperency blobs and process them as white, and put markers on them. These six holes that i draw are always transperency, (gray).

Is there a way to filter them and put markers on them too? Like in picture below?

[DGM]

See if this is what you're looking for. The low contrast with the gray plugs is a bit of an obstacle, so I chose to change the way they're being detected in the last part. The first half is basically unchanged except for the different input files and plug list.

% %%%% PROCESS REFERENCE IMAGE %%%%

% get the image, discard color
reference = imread('goodedgecase.jpg');
reference = rgb2gray(reference);

% detect the plugs
refplug = imfill(imbinarize(reference),'holes');
refplug = bwareaopen(refplug,100);

% remove the plugs from image
refnoplug = reference;
refnoplug(refplug) = 0;

% detect holes
holeth = 20;
maxspecklearea = 100;
solidityrange = [0.7 1];
eccentrange = [0 0.95];
refholes = refnoplug <= holeth;
refholes = ~bwareaopen(~refholes,maxspecklearea);
refholes = bwareaopen(refholes,maxspecklearea);
refholes = bwpropfilt(refholes,'solidity',solidityrange);
refholes = bwpropfilt(refholes,'eccentricity',eccentrange);

% calculate convex hull and object rotation
refhull = bwconvhull(refholes);
S = regionprops(refhull,'minferetproperties');
thref = S(1).MinFeretAngle - 90;

% correct images for rotation
refhull = imrotate(refhull,thref);
refplug = imrotate(refplug,thref);

% close-crop plug mask based on extent of convhull
S = regionprops(refhull,'boundingbox');
rect = round(S.BoundingBox);
xrange = rect(1):sum(rect([1 3]));
yrange = rect(2):sum(rect([2 4]));
refplug = refplug(yrange,xrange);

% stuff needed to create the template
%refholes = imrotate(refholes,thref); % <-- need to carry this to make the template
%refholes = refholes(yrange,xrange); % <-- this is used to manually create the template
%imwrite(refholes,'refholes.png') % <-- write it and open it externally

% get template (this is a label array)
template = imread('holetemplatefull.png');
szref = [size(template,1) size(template,2)];

% find hole centers as linear indices into image
S = regionprops(template,'centroid');
holeidx = round(vertcat(S.Centroid));
holeidx = sub2ind(szref,holeidx(:,2),holeidx(:,1));

% explicitly specify which holes should be filled
% note that i specified 5 and 12 instead of 6 and 11
pluggedholes = [5 12 13 18 19 24 27 29 31 34 35 42 43 45 51 53 56 57 58 61];

% generate logical map of plugs from index list
refplugidx = false(size(holeidx));
refplugidx(pluggedholes) = true;


% %%%% PROCESS WORKING IMAGE %%%%

% get the image, discard color
testimg = imread('rotatedbad2.jpg');
testimg = rgb2gray(testimg);

% detect the plugs
testplug = imfill(imbinarize(testimg),'holes');
testplug = bwareaopen(testplug,100);

% remove the plugs from image
testnoplug = testimg;
testnoplug(testplug) = 0;

% detect holes
holeth = 20;
maxspecklearea = 100;
solidityrange = [0.7 1];
eccentrange = [0 0.95];
testholes = testnoplug <= holeth;
testholes = ~bwareaopen(~testholes,maxspecklearea);
testholes = bwareaopen(testholes,maxspecklearea);
testholes = bwpropfilt(testholes,'solidity',solidityrange);
testholes = bwpropfilt(testholes,'eccentricity',eccentrange);

% calculate convex hull and object rotation
testhull = bwconvhull(testholes);
S = regionprops(testhull,'minferetproperties');
thtest = S(1).MinFeretAngle - 90;

% correct images for rotation
testhull = imrotate(testhull,thtest);
testplug = imrotate(testplug,thtest);
testimg = imrotate(testimg,thtest);

% close-crop plug mask based on extent of convhull
S = regionprops(testhull,'boundingbox');
rect = round(S.BoundingBox);
xrange = rect(1):sum(rect([1 3]));
yrange = rect(2):sum(rect([2 4]));
testplug = testplug(yrange,xrange);
testimg = testimg(yrange,xrange);

% resize moving image to match reference
testplug = imresize(testplug,szref);
testimg = imresize(testimg,szref);

% THIS PART IS DIFFERENT -->
% check hole locations in the working image
% instead of relying on the binary mask (testplug)
% this processes testimg using the blob extents from the template
% this way the plug detection process is independent from the orientation correction
Simg = regionprops(template,testimg,'meanintensity');
observedintensity = vertcat(Simg.MeanIntensity);
% i'm only using the lower threshold where it's necessary due to the low contrast with gray plugs
% at least that retains a bit better robustness for the smaller white plugs
threshold = [125*ones(56,1); 50*ones(6,1)];
testplugidx = observedintensity>threshold;
% <-- THIS PART IS DIFFERENT

% these are linear indices into the template where a plug is missing
plugismissing0 = refplugidx & ~testplugidx; % index into arrays returned by regionprops()
plugismissing = template(holeidx(plugismissing0)); % index into hole template

% these are linear indices into the template where a plug is misplaced
plugismisplaced0 = ~refplugidx & testplugidx; % index into arrays returned by regionprops()
plugismisplaced = template(holeidx(plugismisplaced0)); % index into hole template

% plot the location of missing plugs
imshow(testimg); hold on
[y x] = ind2sub(szref,holeidx(plugismissing0));
plot(x,y,'o','linewidth',3,'markersize',10)
[y x] = ind2sub(szref,holeidx(plugismisplaced0));
plot(x,y,'x','linewidth',3,'markersize',10)

I made a modified test image with plugs in the wrong place. The pluggedholes reference list refers to the plug locations in the original image.

The template label array and the label order have been updated to account for the full 62 holes.

I'm not sure how robust the plug detection will be on the gray plugs if material color or lighting changes much. I picked 50 as the threshold for those locations, but there may be room for adjustment.

[Audrius Bieliunas]

thank u sir, that was what i look for. U save the day.

Answer 2

[Image Analyst]

There are probably lots of ways to solve this but I'd start like this.

1. Threshold to find the large two black rectangles on the right side of the jig.
2. use regionprops to get the two centroids, then find the angle between them
3. use imrotate to make them level.
4. then find the spots again and use imtranslate to put the upper right dark rectangle at some known location. Do that for both images.
5. Then use imabsdiff to subtract the two images.
6. Threshold to find the difference.
7. Call imclearborder to get rid of any differences touching the edge of the image.
8. Call imfill to fill any holes.
9. Call bwareafilt to get only blobs in a certain known area range.
10. Call bwlabel or regionprops to count and locate the missing or extra white blob.

As you can see, pretty easy, it's just that there are a lot of simple steps.

Write back if you can't figure it out.