How can implement parallel pipeline using 3 workers in one computer ?

Dear everyone, Please I have a question:

I would like to used 3 workers (cores) to implement 3 functions in a parallel as pipeline, i.e. the first worker create the data, do some process(fun-add), and sends the results (x) to the first buffer/queue/matrix. Then, the second worker receives the data from the first buffer, do some process (fun-mult), and send the results (y) to the second buffer. The third worker receive the data, do some process (fun-double) and print the output(z). In the following attach the picture show the situation.

My question is that: How can I implement these three functions in parallel pipeline ?

Please, any idea how to solve this problem?

Kind regards Ammar

Answer 1

[Walter Roberson]

You could use spmd and labSend the data from one worker to another.

[Ammar]

Dear Walter Roberson, I would like to thank you for this answer. Please, could you explain more how can I use spmd and labSend. That is because I want to let all the cores work at the same time simultaneously. Thanks

[Walter Roberson]

parpool(3)
spmd
  while true
    if labindex == 1
      data = .... %generate the data
      labSend(data, 2);   %send it to second process
    elseif labindex == 2
      data = labReceive(1);   %wait for initial batch of data
      tdata = .... %transform the data once
      labSend(tdata, 3);  %send it to third process
    else
      tdata = labReceive(2);  %wait for initial batch of transformed data
      newdata = .... %transform the data again
      disp(newdata);
    end
  end
end

[Ammar]

Dear Walter Roberson, I would like to thank you again for this answer. Please, could I ask, how can we save the temporal data in a queue/buffer/matrix ? Then, the next core just make access to the queue to take the data not directly receive from the previous worker. And you can see in the figure the both workers make access to the queue at the same time simultaneously. Thanks - Ammar.

[Walter Roberson]

In order to implement that by using memory instead of a file, you would have to use something like the routine named sharedmatrix from the File Exchange; however, I see a post that claims that routine might not work starting with R2017a. Also, you would have to change the source code for the sharedmatrix routine a little if you are using MS Windows.

There is no built-in routine for handling queues that are based on files. There are a few different approaches. https://www.mathworks.com/help/matlab/import_export/share-memory-between-applications.html shows one of them. That example is not reliable as a Queue, though, as it is not designed to allow there to be several entries in the queue.

It is a bit tricky to get file-based queues working properly; it is common to encounter "race conditions" unless you have operating system support for atomic operations, such as semaphore objects .