Download file using "websave" from a source that requires login via Oauth

How can I download a file from NASA's Archive of Space Geodesy Data at the provided URL, which requires login information and redirection? My current code attempts to download the file directly using websave, but it only retrieves the login page's HTML instead.

https://urs.earthdata.nasa.gov/oauth/authorize

Answer 1

[Chetan]

Hi @tanvi,

It appears that the website you're attempting to access does not utilize the basic authentication protocol.

Instead, it employs OAuth. To download the file, you will need to carry out an authenticated download.

Refer to the websites workflow provided for Authentication:

https://urs.earthdata.nasa.gov/documentation/for_users

This process requires you to obtain an authentication token from the Earthdata Login service, which you then use to configure the HTTP request header for the download.

The following MATLAB code snippet illustrates the steps for this authenticated download:

% Define the URL for the Earthdata Login token service and your credentials
tokenService = 'https://urs.earthdata.nasa.gov/api/users/find_or_create_token';
username = 'yourUsername';
password =  'yourpassword';
credentials = matlab.net.base64encode([username ':' password]);
% Define headers for the token request
headers = struct('name', 'Authorization', 'value', ['Basic ' credentials]);
% Obtain the token
options = weboptions('HeaderFields', {headers.name, headers.value});
tokenResponse = webwrite(tokenService, options);
token = tokenResponse.access_token;
% Now use the token to set the HTTP request header for file download
fileHeaders = struct('name', 'Authorization', 'value', ['Bearer ' token]);
downloadOptions = weboptions('HeaderFields', {fileHeaders.name, fileHeaders.value}, 'Timeout', Inf);
% Define the URL of the file to download and the local filename
fileurl = 'https://cddis.nasa.gov/archive/gnss/data/daily/2021/143/21h/MAR700SWE_R_20211430000_01D_SN.rnx.gz';
filename = 'RINEXFile.rnx.gz';
% Download the file
websave(filename, fileurl, downloadOptions);
ans = '/users/mss.system.qkQ0UA/RINEXFile.rnx.gz'

You can refer to the following MathWorks Documentation for weboptions and websave:

• https://www.mathworks.com/help/matlab/ref/websave.html
• https://www.mathworks.com/help/matlab/ref/weboptions.html
• https://www.mathworks.com/help/matlab/http-interface.html

Hope it helps

Thanks

Chetan