Image Processing: Recognizing arrows and their orientation/direction

Hello, I am trying to write a code that will import an image of an arrow, then tell me what direction it is pointing. The end goal was to eventually mature this into showing a simple small robot with a camera on a track a left or right arrow, and having it move in a given direction. My first problem is deciding how to determine which way it is facing. My first thought was that arrows are usually "heavier" on the side they are pointing, so if i could find the centroid, then show that it was to the right of the midpoint (positive distance), I could say that it was pointing to the right. Conversely if it was a negative distance it was pointing left. So I began with this.

Note: I use %% frequently because I am just beginning with MATLAB and these titles help me remember what each section is doing for me.

%% Clean Workspace
clear all; close all; clc;

%% Import Image
arrow = imread('rightarrowcolor1.jpg');

%% Display Image
figure
imshow(arrow)

%% Convert to Greyscale
img1BW = rgb2gray(arrow);

%% Display Greyscale image
figure
imshow(img1BW)

%% Convert to Binary Image
Binaryarrow = imbinarize(img1BW);

%% Display Binary Image
figure
imshow(Binaryarrow)

%% Locate & Plot Centroid
C = regionprops(Binaryarrow,'centroid');
centroids = cat(1, C.Centroid);

imshow(Binaryarrow)
hold on
plot(imgca,centroids(:,1), centroids(:,2), 'b*')
hold off

It was when I got to this point that I realized I am going to have problems. I see now that depending on the geometry, some arrows won't obey my notion that the pointing end would be "heavier: , or contain a larger surface area, negating the effectiveness of my centroid - offset idea. That oversight got me thinking there is probably a much better way to go about this, so I figured I would reach out. Attached is a simple arrow I began with. Left/Right is all I need for now, but if there is a framework that could expand to Up/Down or angles later on that would be a nice stretch goal.

Answer 1

[Raynier Suresh]

The “regionprops” in MATLAB could give you the orientation angle, to use regionprops the input must be a Binary Image or Label Image or Connected Components.

The below code would give you the angle at which the arrow is facing,

I = imread('arrow.jpg'); % read your image 
Ig = rgb2gray(I); % Convert to gray image 
e = edge(Ig,'sobel'); % Use edge detection to find arrow 
BW = imfill(e,'hole'); % fill the arrow  
K = regionprops(BW,'Orientation'); % find the angle, range of angle is –90 to +90 
angle = round(K.Orientation); 
if angle<0 
    angle = 180+ angle;  % convert the range from 0 to 180, 0 means Right and 180 is left 
end 
angle % Display the angle 

Referring to the following links might be useful:

Regionprops : https://www.mathworks.com/help/images/ref/regionprops.html#d120e215626

Edge Detection : https://www.mathworks.com/discovery/edge-detection.html

Instead of edge detection you could also try image segmentation and based on that you can get the orientation of the arrow using regionprops

[Wiseguy]

Thank you, this was very helpful.

The code as written does seem to struggle with arrows that are at an angle. the one given works, I imported an arrow from google images at a perfect 45 degrees and that was read without issue, but when I try and make arrows using word to try something in between, I get the error

'Error using round

Too many input arguments.'

When using the google image, K returns

K = 

  struct with fields:

    Orientation: 44.9976

so then angle works properly with the round function.

when ran with my arrow, K becomes -

K = 

  308×1 struct array with fields:

    Orientation

So I tried to make another arrow that more closely matches the proportions of the google arrow, and the code seems to like that.

    K = 

  struct with fields:

    Orientation: 55.7975

So, is there anything to learn about maybe longer narrower arrows and why the orientation function has trouble with them? When I talk about this with others I would like to be able to describe more of what goes on "behind the scenes" with that regionprops orientation process and why it seemed to favor the more stubby arrow shape.

google arrow

my arrow

my arrow version 2.

[Raynier Suresh]

The problem is with the image, its pixelated so the edges are not clear and thus it didn't work. one way to solve is to use good quality image and another possible solution is to thicken the edge so that the "imfill" function does its job properly. To thicken the edge you could use the "imdilate" function.

In the below code I have added one line extra from the previous one, this would work for the above image.

I = imread('arrow.jpeg'); % read your image 
Ig = rgb2gray(I); % Convert to gray image 
e = edge(Ig,'sobel'); % Use edge detection to find arrow 
BW1 = imdilate(e, strel('disk',1)); % Thicken the edge
BW = imfill(BW1,'hole'); % fill the arrow  
K = regionprops(BW,'Orientation'); % find the angle, range of angle is –90 to +90 
angle = round(K.Orientation); 
if angle<0 
    angle = 180+ angle;  % convert the range from 0 to 180, 0 means Right and 180 is left 
end 
angle % Display the angle

[Wiseguy]

I had a feeling it was something along those lines. This was tremendously helpful and will be much more robust moving forward. thank you!

[Wiseguy]

One last thing, is that orientation struggles with angle with respect to the arrow head. for example an arrow pointing 160 degrees is also oriented the same as one pointing -20 degres.

[Raynier Suresh]

I suggest you to look at the regionprops('Orientation') and experiment a little with it. 160 and -20 will be the same. Consider a semicircle with theta = 0 will be 0deg and theta = 90 will be +/- 90 deg and theta = 180 will be -0deg this is how regionprops, so arrow at 160deg will be -20deg(I wrote the if loop to change the -20 to 160) . You won't get angle beyond 180deg. Arrow head is not that important.

[Wiseguy]

Ok I see why you added 180 to get a range from 0 to 180, but for my purpose the arrow head IS important, the whole point of the code is to tell me which way the arrow is pointing, left or right, to be expanded to up and down later as well. 45 to -45 is right, 135 to 225 = left, etc..