Parse csv with complex numbers written by Python numpy

I am not actually sure which Python features use this format (I think both numpy and pandas use it), or perhaps this is a part of a larger standard, but basically I have csvs with text that looks like

(-0.0053973628685668375-0.004476730131734169j),(0.005108157082444198-0.005597795916657765j),,,,,,,-298.0,-298.0,37293,-0.7617709422297042,0.7202575393833991,(0.001506298444580933-0.0035885955125266656j)

and I want to parse into a numeric array.

The real-valued scalar entries are easy (well I can do a str2double and it's not super fast but it's acceptable). The blanks are also not too bad because after a simple textscan with a comma delimeter I can find emptys and set to a desired value. But what the heck do I do with these rediculous complex number strings?

There's loopy solutions with regexp or finding the real and imag components but they are too slow when dealing with hundreds of thousands of entries. I could also do things like find entries containing a "j" and process them separately, but is there something better?

[Stephen23]

Please upload a sample data file by clicking the paperclip button.

Note that scipy has functions for saving and loading MAT files, so that might be a worthwhile alternative.

[dpb]

The embarrassing problem with C i/o is it doesn't know anything about complex variables. Why Mathworks hasn't extended MATLAB to deal with them in 40 years is also beyond comprehension. For a scientific language purporting to be user friendly, this is and has always been a glaring oversight.

Ages ago I built a mex function that passed input off to a Fortran mex file with a format string and let the Fortran compiler i/o library that does know complex data types do the parsing. Unfortunately, I lost access to it when left last employer and they had already wiped the machine by the time I thought to ask about it and never got around to rebuilding it.

But

s='(-0.0053973628685668375-0.004476730131734169j),(0.005108157082444198-0.005597795916657765j)';
fmtC='(%f%fj)';
fmt=[fmtC ',' fmtC];
v=reshape(sscanf(s,fmt),[],2).'

v = 2x2

-0.0054 -0.0045 0.0051 -0.0056

<mw-icon class=""></mw-icon>

<mw-icon class=""></mw-icon>

v=complex(v(:,1),v(:,2))

v = 
  -0.0054 - 0.0045i
   0.0051 - 0.0056i

For the full file, one has to go through the very painful with C process of building the format string to match the input record; tedious but doable if the file structure is regular.

Even the new(ish) readmatrix and friends with the import options won't let you set a variable to a complex type. Disgraceful imo. Back when I was still consulting I used to grouse regularly but joy has never ensued....which is why I wrote the mex routine.

Answer 1

[Stephen23]

Forget about fiddling about with strings. Define the parentheses as delimiters and import as numeric:

format short G
M = readmatrix('test.txt', 'Delimiter',{'(',',',')'}, 'ConsecutiveDelimitersRule','join',...
    'LeadingDelimitersRule','ignore', 'TrailingDelimitersRule','ignore')

M = 
  Columns 1 through 6

   -0.0053974 -  0.0044767i    0.0051082 -  0.0055978i         -298 +          0i         -298 +          0i        37293 +          0i     -0.76177 +          0i

  Columns 7 through 8

      0.72026 +          0i    0.0015063 -  0.0035886i

[AB]

Great solution thanks! I didn't realize readmatrix could do things like auto join consecutive delimiters and handle converting text data like "0.005108157082444198-0.005597795916657765j".