How to map multiple calibrated cameras to same coordinate system for 3D world point Projection.

Hello!

I'm working with a set up containing multiple cameras which i want to put into the same coordinate system so that i might project 3D world point onto each respective camera image. My problem is that when i project 3D world coordinates they give image coordinates that are out of frame of my cameras, even if the point projected is clearly visible in the camera images.

I have individually calibrated each camera using Matlabs calibration toolbox and roughly 100 calibration images per camera.

To enable the projection of 3D points i want the CameraMatrix of each individual camera. This i did by taking an image of a calibration board with all cameras to use as the reference board to map my coordinate system. Thereafter i used the board image of each camera together with the calibration to calculate the CameraMatrixes used for 3D world point projection.

Am i doing it the right way? Here is some parts of the code i'm using sadly i cannot provide my calibration data but each individual camera calibration has a re projection error of 0.3 pixels.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

CALIBRATION OF SINGLE CAMERA , i do this procedure for each camera with ImageFileNames containing paths to approximately 100 calibration images, which are taken according to the toolbox recommendations.

% Detect checkerboards in images

[imagePoints, boardSize, imagesUsed] = detectCheckerboardPoints(imageFileNames);
imageFileNames = imageFileNames(imagesUsed);

% Read the first image to obtain image size

originalImage = imread(imageFileNames{1});
[mrows, ncols, ~] = size(originalImage);

% Generate world coordinates of the corners of the squares

squareSize = 2;  % in units of 'millimeters'
worldPoints = generateCheckerboardPoints(boardSize, squareSize);

% Calibrate the camera

[cameraParams1, imagesUsed, estimationErrors] = estimateCameraParameters(imagePoints, worldPoints, ...
    'EstimateSkew', false, 'EstimateTangentialDistortion', false, ...
    'NumRadialDistortionCoefficients', 2, 'WorldUnits', 'millimeters', ...
    'InitialIntrinsicMatrix', [], 'InitialRadialDistortion', []);

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

CALCULATION CAMERA MATRIX:

[imagePoints, boardSize] = detectCheckerboardPoints(ReferenceImage);

squareSize = 2; % in millimeters

worldPoints = generateCheckerboardPoints(boardSize, squareSize);

[rotationMatrix,translationVector]= extrinsics(imagePoints,worldPoints,cameraParams1);

camMatrix1 = cameraMatrix(cameraParams1,rotationMatrix,translationVector);

Plotting each of the cameras using their respective orientation and location gives roughly okey positions but some cameras are off.

orientation1 = rotationMatrix';
location1 = -translationVector * orientation1;

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

I project the 3D points in the following way:

CameraMatrix1=CameraMatrix1'  % Transpose due to having world coordinates in column vector.
ProjPixelcord=CameraMatrix1*Worldcord;

ProjPixelcord=round(ProjPixelcord./repmat(ProjPixelcord(3,:),3,1)); %Dividing by homogeneous coordinate to scale and rounds to get pixel values.

Where Worldcord is a column vector (4x1) with 3D homogeneous coordinates ex: [2;3;-2;1]

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

Im aware i didn't undistort the reference images before i computed the cameraMatrix, this was due to some undistorted images having parts cut of (basically parts at sides turning to black). For the images i did undistort the result did not change and the matrix changed very little.

Any advice would be greatly appreciated!!