I found a solution. Here is the function that I use:
function [max_possible_images] = PIVlab_capture_max_possible_images(ROI,bitmode)
bitmode=16; %even if data from camera is only 10 or 12 bit, the datatype it will be stored in is uint16.
[~,systemview] = memory;
ram_reserve=1; %how much RAM (in GB) is needed for Matlab operations not including image capture
% twice the RAM is needed to use getdata, because getdata creates a copy of the array (why.....!?!?!?)
max_possible_images=floor((mem_free-ram_reserve) / (ROI(3)*ROI(4)*bitmode/8/1024^3 *2)); %max possible images
ROI is the region of interest of the images in x,y, width, height. Bitmode indicates if the camera captures in 8 bit or more. This seems to work pretty good.
The key was to use systemview.PhysicalMemory.Available, which is identical to the numbers that windows taskmanager shows. Only this number seems to make sense, all the other numbers seem to be unrelated to the problem (at least to me).