Reducing a very long vector

I would like to reduce a very long vector without using any (significant amount of) additional memory. For instance, suppose I have slightly over 6GB available and use 6GB of it for a vector A of size [2*N,1]. After some calculations I would like to keep only the first N entries of A. I tried

A = A(1:N, 1);

but it appears that for a short period I will need additional 3GB (which I do not have) as if the first half of A is copied somewhere else and then back. I am looking for a way to move the end-of-array pointer of A to the middle of A and release the second half of A.

Thank you, Borislav

Answer 1

[Walter Roberson]

MATLAB does not provide any explicit way to adjust the sizes.

Your best chance (not guaranteed at all) would be to write an auxillary function,

function A = chopvec(A, N)
  A = A(1:N);
end

MATLAB is able to notice that the input and output array names are the same, and can do some kinds of work "in place" (provided the input array was not a shared one.) I do not know if shortening is one of the things it can do. Possibly not.

Ah.... did you try

A(N+1:end) = [];

?

[Borislav]

Thank you for alleviating my suspicion that I am missing some obvious solution. To summarize my failed attempts, neither

A = A(1:N);

nor

A(N+1:end) = [];

works "in place" (both require additional memory of the order of the left-hand side size). Placing either of them in an auxiliary function does not seem to make any difference (at least on my Windows 7 machine). I have tried both a function in a separate .m file and a function placed at the end of the .m file, where it is used. Thanks

[Matt Fig]

You can write a C-mex to do it in place.

[Walter Roberson]

I'm not sure about that, Matt. There isn't any call to release a part of an allocated data block. You can hack array sizes, yes, but you have the problem of returning the now-unused memory.

There is a chance that if you ask to allocate an empty array, and then hack the array's data pointer to point to the now-unused memory, and hack the array sizes, and then you release the hacked variable, that it just might work, but I sure wouldn't count on it.

[Matt Fig]

I seem to remember Jan Simon doing something like this in an old NG thread. I will try to find it.

[Matt Fig]

I think you are correct, Walter. I found the thread I was remembering and it had nothing to do with this topic. Oh well.

[Jan]

What thread are you talking of? I think it is possible, that I had posted more than one, perhaps two or even three.

Hacking the dimensions inplace is safe (as long as undocumented feature can be). Even with shared variables this does neither crash nor leak memory: mxSetDimension with a smaller number of elements.

But using mxRealloc to release the ununsed memory is a bad idea, when the variable is shared. Although the functions mxUnshare, mxUnshareArray and mxUnreference should cut the connection to a shared variable successfully, I strongly recommend not to apply this in a productive system.

Please send an enhancement request to TMW and ask for the documentation of the sharing mechanism in MEX functions. We all know that the shared-data-copy system is stable and efficient. Therefore it would be a great benefit for MEX programmers to be able, to use this feature for efficiency.