Error using distcomp.remoteparfor - Error writing to output stream

Hello,

I am running a parallel program on a cluster. I am using matlab R2015a. My code calls multiple times a parfor loop. The code looks as follows:

% objects creation, variable allObjs
for iteration=1:100
  %operations on objects allObjs
  parfor i=1:60
    currentObj = allObjs(i);
    evaluate(currentObj);
    allObjs(i) = currentObj;
  end 
  %more operations on objects allObjs after evaluate function
end

Evaluate function does a bunch of operations, files generation, calls a java program and modifies the i-th object. The last line of the parfor loop copies the modified object so I can access the object with the modifications after the parfor loop.

Note that I cannot serialize the parfor as the outer loop is needed to perform some operations before allObjs are evaluated. The program runs fine for the first iterations, however at around iteration 90 it starts to crash. I get the following error:

Error writing to output stream.

    {Error using distcomp.remoteparfor>@(msg)disp(char(msg)) (line 214)
    Error writing to output stream.

    Error in distcomp.remoteparfor>iDisplayStringArray (line 214)
        cellfun(@(msg) disp(char(msg)), cell(msgs));

    Error in distcomp.remoteparfor/displayOutput (line 187)
                iDisplayStringArray(obj.ParforController.getDrainableOutput().drainOutput());

    Error in distcomp.remoteparfor/getCompleteIntervals (line 128)
                        obj.displayOutput();

    Error in evaluate_objects (line 263)
          parfor i=1:100
  }

Could someone please tell me what this error means? is it related to memory? Objects only have around 15 attributes and none of them contain huge data. Maybe the overhead occurs by calling parfor loop many times. Is there a way to optimize of free memory after every parfor? I would appreciate any kind of help.

Thank you,

Answer 1

[Walter Roberson]

If you are not dealing with handle objects then evaluate() is not able to modify currentObject without a transparency violation. If you are dealing with handle objects then allObjs(i) = currentObj afterwards would not be necessary as you would be copying the handle back to itself.... unless, that is, doing the copy back somehow signals to parfor that it needs to retrieve the contents of the handle object.

Other than that, though: it sounds like you are running into memory problems. At the moment I would suspect that it has to do with the object modification, or maybe leaking memory on the java calls.

[Arturo Magana]

Thank you for your answer Walter. Indeed, allObjs are handle objects. I needed to add that last parfor line else the modified object remained as a local copy, therefore I would lose the results after the parfor loop.

I am not an expert in how parfor operates internally but, assuming I have 64 cores, even if I am running the parfor loop many times, the maximum memory parfor loop will use is the size of the object * 60 (and of course the memory used in evaluation). Is that correct? Memory does not somehow accumulate every time parfor is executed right? If memory leaks are not coming from Java programs, then would it make any difference if I create the parpool object every iteration? (other than slower due to the creation of the parpool every iteration)

% objects creation, variable allObjs
for iteration=1:100
  cluster = parcluster('local');
  cluster.NumWorkers = cores-1;
  poolobj = parpool(cluster, cluster.NumWorkers,'IdleTimeout', Inf);
  %operations on objects allObjs
  parfor i=1:60
    currentObj = allObjs(i);
    evaluate(currentObj);
    allObjs(i) = currentObj;
  end 
  %more operations on objects allObjs after evaluate function
  delete(poolobj);
end

Again, thank you for your help.

[Walter Roberson]

This is something beyond my experience.