Assign different resources to different processes running in background

Hi all,

I am trying to set up two independent processes running in the background. The first process would generate data (genData function), while the other would process the data (processData function). The two processes are modular, so I would like to keep them separate. So far, I am sending results from one process to the client, and then to another worker as detailed in this link .

1) is this a good way to have multiple processes running independently, where information might be shared? 2) can I control the resources (# of workers) I can allocate to each process? Should I use instead a backgroundpool? Or does MATLAB somehow allocate the resources the best way possible?, 3) can I get afterEach to run in the background? thank you

% First, create a parallel pool if necessary
if isempty(gcp())
    parpool('local', 1);
end
% queue to generate data
q1 = parallel.pool.PollableDataQueue;
% Get the worker to construct a data queue on which it can receive messages from the client
workerQueueConstant = parallel.pool.Constant(@parallel.pool.PollableDataQueue);
% Get the worker to send the queue object back to the client
workerQueueClient = fetchOutputs(parfeval(@(x) x.Value, 1, workerQueueConstant));
% Generate data in one worker
future1 = parfeval(@genData, 1, q1);
% Get the worker to start waiting for messages
future = parfeval(@processData, 1, workerQueueConstant);
%x = [];
for idx = 1:100
    %x = [x, randn(1)];
    x = poll(q1);
    send(workerQueueClient, x);
    %send(workerQueueClient, rand(1));
end
% Send [] as a "poison pill" to the worker to get it to stop
send(workerQueueClient, []);
% Get the result
fetchOutputs(future)
% Functions
% Process data
function out = processData(qConstant)
q = qConstant.Value;
out = 0;
while true
    data = poll(q, Inf);
    if isempty(data)
        return
    else
        out =  data;
    end
end
end
%Generate data
function x = genData(q1)
x = [];
for jj = 1:10
    x = [x, rand(1)];
    send(q1,x);
end
end

Answer 1

[Vidip]

I understand that you are trying to set up two independent processes running in the background. The first process would generate data while the other would process the data. Using parallel computing in MATLAB for running independent processes is a good approach, especially when dealing with parallelizable tasks.

The approach you've taken, using ‘parfeval’, is suitable for running two processes independently. It allows for communication between them using queues, providing a form of data exchange.

MATLAB allows you to control the number of workers in a parallel pool using the ‘parpool’ function. In your case, you've set it to 1 worker (parpool('local', 1)). You can adjust this number based on your system's capabilities.

MATLAB typically allocates resources based on available CPU cores. If you want more fine-grained control, you might consider using a ‘backgroundPool’ where you can specify resource requirements for individual tasks. And the ‘afterEach’ function is typically used with ‘parfor’ loops, and it's designed to run on the client (not in the background on the workers).

For further information, refer to the documentation links below:

• https://in.mathworks.com/help/matlab/background-processing.html
• https://in.mathworks.com/help/matlab/ref/parallel.backgroundpool.html
• https://in.mathworks.com/help/parallel-computing/parpool.html