Fast conversion of 2 matrices to 1 complex matrix

Hi,

After reading my data from an external binary file, I have a 3 dimensional array representing one complex matrix, like this:

A(1,:,:) = rand(10);    % real part
A(2,:,:) = rand(10);    % imaginary part

I need to convert that into a complex matrix, e.g.

A = complex(A(1,:,:), A(2,:,:));

But this is rather slow. As far as I have been able to find out, Matlab stores complex matrices internally as two matrices, one for the real and one for the imaginary part. This is about the same as my original 3-dimensional matrix -- and should be quite fast! My assumption therefore is that Matlab uses a temporary variable which of course needs to be allocated, resulting in slow code.

Is it possible to make Matlab do this conversion without allocating more memory? Maybe just setting the 'complex' attribute to A... It might be possible using mex files, but I never used them before, and I would prefer a 'pure' Matlab solution.

Any comments, hints or keywords I can google for are welcome!

Regards Argon

Answer 1

[James Tursa]

For your specific example, the A(1,:,:) and A(2,:,:) data blocks are contiguous memory and, more importantly, were both allocated with a single malloc (or similar) call in the background. They cannot be legally separated into separate real and imaginary parts of another variable. You will be required to copy the data as you (and Wayne King) are doing.

Regarding mex routines, there are ways to do stuff like this. I have code (as yet unpublished to the FEX) that can pretty much do what you are asking, but it involves manipulating the mxArray structure behind the scenes (i.e., not using the official API functions). Also, for your particular case, one would have to keep a shared data copy of the new complex variable locked up in the mex routine to prevent MATLAB from trying to free the imaginary part (if that ever happened MATLAB would crash). To clear it you would have to manually reverse the process that you did to create it. So it gets to be very tricky to manage, but technically it can be done. Can you redo the part of your code that creates A and instead put the A(1,:,:) and A(2,:,:) parts into separate variables from the outset? If so, then what you are asking becomes much easier to do in a mex routine (but still requires unofficial behind the scenes techniques).

[Argon]

So there is no easy way without mex. Reading in the real/complex parts separately would probably result in reading the file twice (it's in bip interleaved format) so I expect that to be quite slow as well. I might try to read the data in chunks such that I can convert it to complex notation chunk by chunk. I don't want to dive into the mex business if I don't have to, but it's nice knowing that there is a way if I need it one day.

[James Tursa]

CORRECTION to my earlier post: The A(1,:,:) and A(2,:,:) slices are not in fact contiguous in memory, they are interleaved (as you stated). So there is no way to do this in a mex function either. The real and imag parts would have to be in the A(:,:,1) and A(:,:,2) slices for this to be possible even with a mex function. But that is assuming that the data is already in memory.

If you are reading the data from a file, then you are in luck if we are talking about a regular binary file and not a .mat file. It would not be hard at all to do the reading in a mex routine and de-interleave the data one element at a time as you are reading it and stuff the real and imaginary elements separately into the appropriate spots in a MATLAB variable as you go. Do you know (or can you easily find out) the size of the variable prior to reading the file?

[Argon]

I do -- and it is a regular binary file. I've thought of doing just that out of a Matlab function, but I assumed that calling fread for every value separately would be slower than calling it once just because of the function overhead. Maybe I should do some tests here...

[James Tursa]

Haven't done the comparison myself yet, but my guess is that doing this at the Matlab level would in fact be quite a bit slower than doing it in a mex routine because of all the extra overhead involved in creating/copying/deleting the individual Matlab variables involved. I.e., when you fread at the mex level you essentially get just the data. But when you fread at the Matlab level you have to create these 100+ byte Matlab variable structures to hold each individual value. All that overhead will be quite a drag on performance. If you need help setting up the C-code for this let me know.

Answer 2

[Wayne King]

A(1,:,:) = randn(100);
A(2,:,:) = randn(100);
B = squeeze(A(1,:,:)+1i*A(2,:,:));

Seems pretty fast to me, is your matrix really big?

[Argon]

It is essentially a (complex) image, so it can be quite large, from 10'000 x 10'000 up to 10'000 x 100'000 (complex) pixels, so I really don't want it to be in memory twice at any point.

Answer 3

[James Tursa]

See this FEX submission for reading and writing interleaved complex data in R2018a or later without extra data copies:

https://www.mathworks.com/matlabcentral/fileexchange/77530-freadcomplex-and-fwritecomplex

See this FEX submission for reinterpreting an existing real variable as interleaved complex or vice-versa in R2018a or later:

https://www.mathworks.com/matlabcentral/fileexchange/65842-sharedchild-creates-a-shared-data-copy-of-contiguous-subset

[Muhammad]

Hi James,

I have the signal in complex format, I want to convert it to .dat form but not interleaved. I tried the fwritecomplex but it didn't help as it interleaved the data. I want to save each complex value as single value not interleaved. Is it possible?

Thanks in advance.

[James Tursa]

Can you clarify what you want to do? Preferably with a very small example? Show exactly what you are starting with and exactly what you want as an end result. E.g., if you want data written to a file show excatly how you want it written to the file with the example data. Also what MATLAB version are you running with? The approach is very different depending on whether you are using R2018a & later vs R2017b & earlier. Thanks.