What are n and m? Is either greater than 65500? Should it be?
I don't know if using factor() this way is really doing what you want it to. If you're trying to reshape the vector into a roughly square array, this isn't going to do that. Factor() returns the prime factors, all of which are likely very small compared to the square root of numbytes. If your goal is to make the image reasonably square so that writing/viewing it is more practical, then consider something like this:
numbytes = 600; % for example
f = 1:ceil(numbytes/2);
f = f(rem(numbytes,f)==0);
f = sort([f; numbytes./f],1,'ascend'); % factor pairs
ar = f(1,:)./f(2,:); % normalized aspect ratio
[~,idx] = max(ar); % find AR closest to 1
imagegeometry = f(:,idx) % this is the target image geometry
If the particular value of numbytes doesn't factor well enough to produce an image with an aspect ratio reasonably close to 1 (i.e. maximizing available area within the geometry constraints of the filetype), then consider padding the vector such that it factors more favorably.
The geometry limitations of other filetypes may differ. PNG seems to write images up to 1E6px wide. TIF supports at least 500E6px, though you'll probably run into filesize limits before then if your aspect ratio is sensible.
