Pass Array Variable to MATLAB Function in Bash

Aim

On a Linux High Performance Computing (HPC) system, I am using multiple text files to set variables and array variables in a bash script. These are variables are then the input arguments to a MATLAB function that is run through bash. This MATLAB function runs iteratively creates url download strings (based on the variables mentioned above) and then download corresponding temperature data using MATLAB's websave function. Notably, the MATLAB script uses parfor to parallelise a for loop and spread it over 44 cores on a Linux high performance computing cluster. The MATLAB functions runs fine on the laptop, but extremely slowly, that's why I need to run it from the High Performance Cluster.

Issue

If I open the matlab app via the linux commandline then I can get the function to run. However, I want to send this as a job to a HPC scheduler - meaning I cannot work interactively. Thus, I need to run the function from a bash script.

I am having trouble passing the bash script variables to the matlab function. Note I have tried removing the single quotes from around the variables in the matlab function call, however I get the same error.

Function Example

function f_Download_CRW_Temp(Lat, Long, StartYear, EndYear, SiteNumber, NumCores, FileType)
% <code to set the download url string based on site, latitude, longitude, and start and end dates>

parpool(NumCores) % this will be 44 in the 
parfor d = 1:length(Years)
    options  = weboptions('Timeout',Inf);
    url      = % <code>
    filename = % <code>
    try
        websave(filename,url,options);
    catch
        % < Failure report code >
    end
end
% < write failure report to text file>
end

Bash Script (not in MATLAB syntax)

cd CRW_Temp_Profiles

# Set Variables
IFS=$'\r\n' GLOBIGNORE='*' command eval  'LatArray=($(cat Lat.txt))'
IFS=$'\r\n' GLOBIGNORE='*' command eval  'LongArray=($(cat Long.txt))'
IFS=$'\r\n' GLOBIGNORE='*' command eval  'SitesArray=($(cat Sites.txt))'
Site=$(echo ${SitesArray[${SLURM_ARRAY_TASK_ID}-1]})
FileType=".csv"

# Check Variables were read properly
echo 'Site is Number '${Site}
echo 'Row-specific LatLons are:'
echo '    deg N = '${LatArray[${SLURM_ARRAY_TASK_ID}-1]}
echo '    deg S = '${LongArray[${SLURM_ARRAY_TASK_ID}-1]}
echo ' All Lats are'
echo ${LatArray[*]}

# run MATLAB function
module load MATLAB
matlab -nodisplay -nojvm -nodesktop -nosplash -r "try; f_Download_CRW_Temp('$LatArray', '$LongArray', "1985", "2019", '$Site', "44", ".csv"); end; quit;"
module purge

Bash Output and Error Message

Site is Number 1
Row-specific LatLons are:
    deg N = -16.5
    deg S = 149.8
 All Lats are
-16.5 -20 -19.5 -11.5 -23.5 -21.5 -31.5 ..... <100 entries of site latitudes> .... 11.2 11.2

                                                                                          < M A T L A B (R) >
                                                                                Copyright 1984-2018 The MathWorks, Inc.
                                                                                R2018b (9.5.0.944444) 64-bit (glnxa64)
                                                                                            August 28, 2018


For online documentation, see https://www.mathworks.com/support
For product information, visit www.mathworks.com.

 try; f_Download_CRW_Temp('-16.5', '149.8', 1985, 2019, '1', 44, .csv); end; quit;
                                                                                   
Error: Invalid use of operator.

Answer 1

[Liam Lachs]

I found my answer.

When you are putting string variables in the matlab function as arguments, in matlab they would appear in double quotes "", but in the 1-line script the entire matlab call is in double quotes, so you must also escape the argument-specifc double quotes with "\". So the code looks like:

matlab -nodisplay -nodesktop -r "f_Download_CRW_Temp('$LatArray', '$LongArray', \"1985\", \"2019\", '$Site', \"44\", ".csv"); quit;" 

Note that I dumped the -nojvm and the -nosplash options as they were causing unexplained errors in my parallelised function.

[Raymond Norris]

Hi Liam,

-nodisplay implies -nodesktop. Also, the prefered choice (since R2018b? don't quote me) is to use -batch instead of -r. And with -batch you can drop the "quit".

Raymond

Answer 2

[Steven Lord]

The getenv function may be of use to you.