Is there a way to ensure MJS schedules workers on different servers instead of filling up each server one by one?

2 vues (au cours des 30 derniers jours)
I have been using Matlab Parallel Server and MJS to split up a large time consuming process over several servers. The process does not use a lot of computing power, but cannot be parallelized except by creating additional workers (I am using Matlab to run a 3rd party software using an SDK).
This means that I need to have multiple workers on each server in order to perform the process faster.
Each additional worker per server does decrease my total time, but there are diminishing returns.
What would be ideal would for the jobs to be spread as evenly across the servers as possible, but what I am seeing is that jobs fill up workers on each server sequentially.
i.e. if I have 60 jobs, 6 servers, 90 workers as 15 workers per server.
I am seeing:
Server 1: 15/15
Server 2: 15/15
Server 3: 15/15
Server 4: 15/15
Server 5: 0/15
Server 6: 0/15
Instead of:
Server 1: 10/15
Server 2: 10/15
Server 3: 10/15
Server 4: 10/15
Server 5: 10/15
Server 6: 10/15
I've thought to create an individual cluster for each server, and then send jobs sequentially to each cluster, but I don't know if this will create problems at the head node. It definitely will make result acquistion and error logging more difficult.
Is there some other way to set this up?
Thanks

Réponses (1)

Alvaro
Alvaro le 26 Jan 2023
There might be a workaround with the startworker and stopworker scripts.
Alternatively, you could try using a different scheduler.
In the meantime, I have forwarded to the relevant team your interest in being able to schedule jobs to specific nodes in the cluster.

Catégories

En savoir plus sur MATLAB Parallel Server dans Help Center et File Exchange

Produits


Version

R2022a

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by