Synchronization for parallel code
3 vues (au cours des 30 derniers jours)
Afficher commentaires plus anciens
Hi,
I have a simple task to accomplish using the parallel tool box and distributed computing server. I need to execute a program (unix() call) on multiple workers. The workers run the same program but the program takes in a data file and writes a data file. Each worker's running program takes a different data file in a directory which all workers have access to. There are more data files than workers, so I would like a worker to choose another data file to run after executing the program on a previous data file. I want all workers to do this until all data files are exhausted. After reading the doc, it seems spmd is the way to go.
I have a cell array of file names and I would like to create an array of booleans of the same dimension to use for synchronization. When a worker accesses a file to work on, it should set the boolean at the array index of the index of the file that it's working on. When each worker finishes it's program run, it indexes through the booleans to find a file that has not been run and sets the boolean for that file index and then runs that one.
Is there any way for the workers to see a common array and also modify this same array?
Right now I'm doing it in a parfor loop that is dimensioned for maximum number of workers. This works fine, but I have a lot more files than workers. Therefore I have to wait for the parfor loop to finish before manually repeating the process for the remaining files. The program takes variable hours to complete on each worker so the parfor loop is bottlenecked by the longest run.
Is there a way to accomplish this using tasks as well? It seems like a simple thing but I have not found any examples of how to do this.
Thank you for reading.
0 commentaires
Réponse acceptée
Jill Reese
le 17 Oct 2012
It sounds like you have independent tasks with more work to do than workers to do it. You also would like the tasks to be properly load-balanced. With these goals, parfor is the right tool. Note that parfor is a different construct than spmd in the Parallel Computing Toolbox.
Since you already have a cell array of file names and all the workers have access to the directory they are in, parfor can easily help you to load data from the files and use the data.
For example, if a different A variable is saved in each file, this code will allow 2 workers to process 5 files in an independent fashion without stomping on each other (no boolean array required).
matlabpool open local 2
inFiles = {'A.mat','B.mat', 'C.mat','D.mat','E.mat'};
outData = cell(1,length(inFiles));
parfor i=1:length(inFiles)
S = load(inFiles{i}, 'A');
outData{i} = S.A+3;
end
At the end of the computation, each worker has stored the result of its computation in a cell array with the same number of elements as the number of files processed.
Unfortunately, you cannot call save from within a parfor loop. You didn't mention how much data you need to save at each iteration. It may be that you can store the results as shown above until the parfor loop completes and then have a post-processing save step that executes outside of a parfor loop.
3 commentaires
Jill Reese
le 18 Oct 2012
The parfor command automatically disperses the work (the total number of inFiles in your example) amongst the available workers. There is no guarantee that worker 1 will be assigned iteration 1. In fact, there is no guarantee that workers will execute their assigned values of i in any particular order. This is one reason why all iterations must be completely independent.
Once a worker completes the first iteration that it was assigned, it moves on to the next iteration that it was assigned to execute. This continues until all the work is complete. Does this help?
Plus de réponses (0)
Voir également
Catégories
En savoir plus sur Parallel for-Loops (parfor) dans Help Center et File Exchange
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!