Batch scheduler or pick up the run after wall time expires or checkpoints

Hi,

I have a matlab program which I submitted as sbatch file. However, the walltime for matlab license on supercomputer to which I submitted expires after 24hrs. My data is saved at each step so I am not worried of losing the data at any point. I am trying to find a way to start running the file after it left off on the for loop. For example, considering I am running a time program like, if my code ends at j = 45000 and j = 30000 after 24hrs, how should I restart again at i = 45001 and j = 30001. Any hints ?

for i= 1:100000
for j = 1:60000
code....
save(data)
end
end

Answer 1

[Jan]

DataFile = 'C:\Temp\YourData.mat';
if isfile(DataFile)
    Data = load(DataFile);
else
    Data.i0 = 1;
    Data.j0 = 1;
end

for i = Data.i0:100000
    Data.i0 = i;
    for j = Data.j0:60000
        Data.j0 = j;
        
        Data.value = code...
        save(DataFile, 'Data');
   end
end

Saving the file 6'000'000'000 times will waste a lot of time. Then this will loose the last inner loop, but is most likely much faster:

DataFile = 'C:\Temp\YourData.mat';
if isfile(DataFile)
    Data = load(DataFile);
else
    Data.i0 = 1;
end

for i = Data.i0:100000
    Data.i0 = i;
    for j = 1:60000
        Data.value = code...
    end
    save(DataFile, 'Data');
end

A smart option would to store the file every minute only using clock and etime. Then you loose less then a minute per day, but need only the time for 1440 save commands.

[nlm]

There is no pre-allocation of memory like data.mat. I just writing the data into .bin files. Can you suggest something for my code below ?Thanks.

%%%%%%% NDVI Processing %%%%%%%%%%%%%%
%%%%% Code by Maheshwari Neelam %%%%%% Date: 12/07/2020
clear all
close all
clc
addpath('path')
filename = 'path/ocean_mask.mat';
mask = matfile(filename);
D=dir('./20*');
Nobs=size(D,1);
date_fixed = (1:1:366)' ;
num_row = 1000; % rows per group
% Initialize day counter
clear time
k=1;
band_ID  = 1 ;
lat_band = (1:num_row)+(band_ID-1)*num_row;

ND = 366;
tmp = 'path/data/';
bnd = '/data_lat10.bin';
fids = zeros(1,ND);
for day = 1 : ND
    fids(day) = fopen(sprintf('%s%d%s', tmp, day, bnd), 'a');
end

tic
for lat = 1:length(lat_band)
    tic
    for lon = 1:36000
        if (mask.ocean_mask(lat_band(lat),lon)~=9)
          data_1km = code;
            end
        else
            data_1km = ones(366,1)*9;
        end
        for day = 1:ND
            fwrite(fids(day), data_1km(day), 'double');
        end
        clear  data_1km day
    end
    toc
end
toc
fclose(fids)

[Jan]

Removing the data by the clear commands is usually a waste of time in Matlab.

Adding a folder to the path only to access the local files is a bad design. Only Matlab's functions should be part of the path, but there is no reason to expand the path for accessing data files. Use absolute pathnames instead.

I cannot run your code, but it looks like the part "mask.ocean_mask(lat_band(lat),lon)" might consume a remarkable part of the processing time. Do you have good reasons to access the MAT file extremely often? Is it far too large to match into the RAM?

"There is no pre-allocation of memory like data.mat." - I've showed you a method how the loop indices can be stored such, that restarting the script starts the loops at where they have been stopped bevor. This is not a pre-allocation. Nevertheless, this method cannot work reliably in your case, because you append data to open files. If the schedular of the cloud system cancels your process brutally, there is no guaratee, that the open files are left in an accurate state. There is no way to modify your code to let it be stoppable.

As far as I can see, you need a completely different approach. The idea of opening 366 file simultaneously and the time consuming access of the MAT file might be the main problem. If "code" does not hide any huge computations, the code should run fluently in far less then 24 hours. So instead of changing the code to accept a hard stop after 24 hours computing time, it is much better to improve the code to run efficiently. Currently it seems to spend the time with stressing the disk only.