Finding Transformation Matrix of a viewer3d object

I need to map 3D vertices in a viewer3d object to 2d figure coordinates. It's actually because I want the user of my app to be able to select a set of points by drawing a polygon, as I have mentioned here.

The viewer3d has some camera properties: cameraPosition, cameraTarget, cameraUpvector and cameraZoom.

But appearently I cannot use these parameters to construct a transformation matrix, since the camera zoom seems to be relative; it's 1.0 at the beginning of the display and is not a function of the window size, object dimension etc.

Is there any way I could extract the camera information needed?

[Catalytic]

What is "the beginning of the display"? The lower left corner?

[Reza]

Yes, that would work. Any mapping to coordinates on the figure and I'm good to go!

[Matt J]

It's actually because I want the user of my app to be able to select a set of points by drawing a polygon

If so, a mapping from 3D to 2D is not what you would need. You need the other way around. A polygon is 2D, so the input to your mapping would be a 2D selection and the output would be 3D locations on the object.

[Reza]

Yes, that could be the case. I was thinking of calculating the position of my 3D points to the 2D coordinates and see which ones resides inside the polygon. But it could also be the other way around (maybe more complicated though)

Answer 1

[Matt J]

I think the 3x4 camera projection matrix P would be,

h=gca();
camz=h.CameraTarget-h.CameraPosition;
camy=-h.CameraUpVector;
camx=cross(camy,camz);

P = normalize([camx;camy;camz],2,'n',2) * [eye(3), -h.CameraPosition(:)];

This P should map things from 3D data units to 2D coordinates on the plane shown in Graphics Camera Terminology. I will call this 2D coordinate space "GCT".

Using this, you should be able to find the coordinates of the "corners" of the image rectangle enclosing the 3D axes in GCT coordinates:

C=table2array(combinations(xlim,ylim,zlim))'; %3D corners

 fn=@(q) q(1:2,:)./q(3,:); %resolve homogeneous projection
 
cmap= fn( P*[C;ones(1,8)] ); %3D corners mapped to GCT

upperLeft=min(cmap,[],2);  %upper left corner of plot box in GCT coords
lowerRight=max(cmap,[],2); %ower right corner of plot box in GCT coords

Finally, getframe() produces an image whose corners, in pixel coordinates will correspond 1-1 with upperLeft and lowerRight, so you can use that to map your drawpolygon() vertices from image pixel coordinates to 2D GCT coordinates.

So, doing all of that will map your viewer3D coordinates and your polygon vertex coordinates to a common 2D coordinate system.

[Reza]

Thank you for your comprehensive response.

This looks very close to what the solution might be. I'm trying to make it work with some modifications. Haven't succeeded yet. Right now the mapped points seem to be streched in one direction.

There might be an implementation error on my side, I will check and update on that.

Thanks again :-)

p.s: I made two functions based on your advice and here's what it looks like:

function [P, upperLeft, lowerRight] = findProjection(vr, pc)
% Finds the projection parameters based on the parameters of the viewer3d
% object vr and the pointCloud displayed in it, pc.

camz=vr.CameraTarget-vr.CameraPosition;
camy=-vr.CameraUpVector;
camx=cross(camy,camz);

P = normalize([camx;camy;camz],2,'n',2) * [eye(3), -vr.CameraPosition(:)];

% Find the range of point values
xlim = [min(pc.Location(:, 1)), max(pc.Location(:, 1))];
ylim = [min(pc.Location(:, 2)), max(pc.Location(:, 2))];
zlim = [min(pc.Location(:, 3)), max(pc.Location(:, 3))];

C=table2array(combinations(xlim,ylim,zlim))'; %3D corners
 fn=@(q) q(1:2,:)./q(3,:); %resolve homogeneous projection
 
cmap= fn( P*[C;ones(1,8)] ); %3D corners mapped to 2D
upperLeft=min(cmap,[],2);  %upper left corner of plot box in GCT coords
lowerRight=max(cmap,[],2); %lower right corner of plot box in GCT coords
end

function p2d = project2d(pc, P, upperLeft, lowerRight, ax)
% Project 3D points in the pointCloud pc to 2D coordinates of UIAxes ax

fn=@(q) q(1:2,:)./q(3,:);
Locs = [pc.Location'; ones(1, pc.Count)];  
p2d = fn(P*Locs)';

ax.XLim = [upperLeft(1), lowerRight(1)];
ax.YLim = [upperLeft(2), lowerRight(2)];
axes(ax, 'equal');
% This is where there might be a problem with axis aspect ratio,
% Although the points are streched even when viewed in another figure
end

[Reza]

So after hours of trying, I finally got to solve the problem. Here's some aspects that needed consideration:

• The CameraUpVector was not really normal to the view direction, it only pointed upwards. So camx and y and z needed to be calculated a bit differenty (this is why the image was streched in one direction):

camz=vr.CameraTarget-vr.CameraPosition; 
camx=cross(vr.CameraUpVector,camz);
camy = cross(camz, camx);
P = normalize([camx;camy;camz],2,'n',2) * [eye(3), -vr.CameraPosition(:)];

• In finding the upperLeft and lowerRight points, the camera target must stay in the center:

pt = vr.CameraTarget';
targetMap = fn( P*[pt;ones(1,size(pt, 2))] );
upperLeft= targetMap - max(abs(cmap-targetMap),[],2); %upper left corner of plot box in GCT coords
lowerRight=targetMap + max(abs(cmap-targetMap),[],2); %ower right corner of plot box in GCT coords
% Also to make sure the center stays on the target

• The viewer3d camera seems to be orthographic, so it only worked when the camera was positioned far away. I'm sure this would be unnecessary if I considered that in the formulations.
• To map the points to axis coordinates, the aspect ratio of the axes and points should be considered.
• Also there was a ratio, which seems to be applied to keep some margins for the figure. I found this to be 0.9 experimentally.
• When rotating the figure, the zoomings and other calculations should not be done again, so it should only be calculated once at the beginning. Here's how I did all that:

function [p2, ratio] = map2fig(p2d, upperLeft, lowerRight, X, Y, ratio, zoom)
% ratio is empty at the beginning. zoom is the cameraZoom parameter which
% is 1 at the beginning.

center_data = (upperLeft+lowerRight)'/2;
center_ax = [X, Y]/2;

% Once the ratio is calculated, changing the camera properties doesn't
% change it. So the ratio must be calculated just once.
if(isempty(ratio))  
    dim = upperLeft-lowerRight;
    ar_data = dim(2)/dim(1);    % Aspect ratio for the data
    ar_ax = Y/X;                % Aspect ratio for the figure
    r = 0.9; % Experimentally, this seems to be a ratio applied for margins
    
    if(ar_data>ar_ax)   % data should be normalized based on figure height
        ratio = Y/dim(2)*r;
    else                % data should be normalized based on figure width
        ratio = X/dim(1)*r;
    end
end
p2 = (p2d-center_data)*ratio*zoom+center_ax;
end

Some images of how it worked:

Initial viewer3d image

Initial Alignment

After Some Rotation and Zoom

So it seems that the problem is finally solved! Thank you.

Do I need to provide full details or add another answer for others to see?

[Matt J]

Do I need to provide full details or add another answer for others to see?

What you have is fine, I think.

I'm a bit surprised that it matters, though. Even if the up vector is not normal to the view direction, I would still expect that the transformed polygon would enclose the transformed points, even if the dimensions are distorted, which is the only thing your app requires.

[Reza]

I'm not going to show the points to the user. The user only sees the nice viewer3D rendered image, then selects some parts of it for edits. That's why it was important to figure out the mapping precisely.

Everything needs to look professional, the app should have the feeling of a final product.

Actually I was surprized myself how I could create a professional looking app using matlab!

Answer 2

[Catalytic]

Maybe you could use estimateCameraParameters using a list of hand-picked points.

[Reza]

Thanks for your answer, that could certainly be helpful while I try to figure out how this system is working.

I'm wonderring how that could be eventually, since I'm developing an app which would be used by the end user. I want the user to be able to view their 3D object, and select a set of points to remove or modify. So it would be difficult to ask them to handpick some points. Or maybe I could do that automatically?

I was hoping there would be a way to extract usable camera properties for this task.