webread with authentication returns HTTP error 400

I am attempting to use webread to download data from a REST web service that uses authentication. I am having problems and I think it is with the authentication. I can put the following into a web browser and get the data I want to appear in the browser:

http://dummyserver/AQUARIUS/Publish/AquariusPublishRestService.svc/GetAuthToken?user=dummyname&encPwd=dummypassword

I get a successful token like: 9fab1c9fee2a18bd6c2b889595d61131. Then I put the following into the web browser

http://dummyserver/AQUARIUS/Publish/AquariusPublishRestService.svc/GetTimeSeriesData?dataId=QRiv.Derived-Working-DayMean@13235000%20S%20Fk%20Payette%20Riv%20at%20Lowman,%20ID&view=Public&queryFrom=2014-09-01%2000%3A00%3A00&queryTo=2014-09-02%2000%3A00%3A00&tok=9fab1c9fee2a18bd6c2b889595d61131

This gives me the data that I expect (a text file in the web browser). I turn this into Matlab code:

usr = 'dummyname';
pwd = 'dummypassword';

url = 'http://dummyserver/AQUARIUS/Publish/AquariusPublishRestService.svc';

dat = '/GetTimeSeriesData?dataId=QRiv.Derived-Working-DayMean@13235000%20S%20Fk%20Payette%20Riv%20at%20Lowman,%20ID&view=Public&queryFrom=2014-09-01&queryTo=2014-09-02';

urltok = [ url, '/GetAuthToken?user=', usr, '&encPwd=', pwd ];

tok = webread( urltok )

tok = 8cef334c4b48f452634fdf3ee8208a4b      % this works

urldat = [ url, dat, '&tok=', tok ]

data = webread([ urldat ])    % *this does not work*

And I get the following response from the last webread()

Error using readContentFromWebService (line 46)
The server returned the message: "Bad Request" for URL,
'http://dummyserver/AQUARIUS/Publish/AquariusPublishRestService.svc/GetTimeSeriesData?dataId=QRiv.Derived-Working-DayMean%4013235000%20S%20Fk%20Payette%20Riv%20at%20Lowman%2C%20ID&view=Public&queryFrom=2014-09-01&queryTo=2014-09-02&tok=8cef334c4b48f452634fdf3ee8208a4b'
(with HTTP response code 400).
Error in webread (line 122)
[varargout{1:nargout}] = readContentFromWebService(connection, options);
Error in get_aquarius_test (line 51)
data = webread([ urldat ])

From what I have read the 400 error code means that the request was malformed. However, I can't figure out what is malformed about the request.

Can anyone tell me how to check what Matlab is actually sending to the server so I can try to determine how it is malformed? Or can you give me other advise on how to solve this.

Other information: The Aquarius server is internal to my company and is on our own computers. I have tried different web browsers and they all seem to work fine, just Matlab has the problem. The Aquarius server stores hydrologic information and is from http://aquaticinformatics.com/.

[Shahd Ewawi]

Have you found an answer for this question? I have the same problem.

[Mark Morehead]

No. I got busy solving other problems. I need to get back to this at some point.

[Luis Galindo]

Any answers to this yet?

Answer 1

[Tushar Athawale]

Please specify the parameters for your URL using "weboptions" in MATLAB. For example, you will need to call "webread" as follows:

>> options = weboptions;
>> options.Username = '<username>';
>> options.Password = '<password>';
>> options.ContentType = 'json';
>> options.RequestMethod = 'Get';
>> URL = 'https://domain.com/path/to/data';
>> response = webread(URL, options)

[Mark Morehead]

Our server requires token authentication, so the lines:

urltok = [ url, '/GetAuthToken?user=', usr, &encPwd=', pwd ]
tok = webread( urltok )

Have the username and password in them instead of the weboptions... So I have not directly called weboptions and I am using the defaults. I did try using various weboptions, but did not succeed. When I type options=weboptions, I get:

options = 
  weboptions with properties:

      CharacterEncoding: 'auto'
              UserAgent: 'MATLAB 8.6.0.267246 (R2015b)'
                Timeout: 5
               Username: ''
               Password: ''
                KeyName: ''
               KeyValue: ''
            ContentType: 'auto'
          ContentReader: []
              MediaType: 'application/x-www-form-urlencoded'
          RequestMethod: 'auto'
            ArrayFormat: 'csv'

Thank your for the help

[James Belanger]

I had a similar question about encoding the token but the REST web service I was using required the token in the header. The solution was to utilize the weboptions function and specify 'token' for 'KeyName' and the token value for 'KeyValue'. Example:

opt = weboptions('KeyName','token','KeyValue','xxxxxxxxxx');

url = 'http://www.ncdc.noaa.gov/cdo-web/api/v2/data?datasetid=GHCND&locationid=ZIP:28801&startdate=2010-05-01&enddate=2010-05-01';

data = webread(url,opt);

[YH]

Similar to above approach would be to pass in the access token to HeaderFields. I had similar issue using Oauth2 and the below worked:

headerFields = {'Authorization', ['Bearer ', access_token]};
options = weboptions('HeaderFields', headerFields, 'ContentType','json');
data = webread(url, options);

[Aminu bugaje]

data_forecast = webread('https://api.solcast.com.au/world_pv_power/estimated_actuals?latitude=-0.436791&longitude=34.206028&capacity=30&tilt=12&azimuth=0&loss_factor=0.9&format=csv&api_key=Q7pW-PeeTU6yYc3yHd3MwWNOJf627_Hc')

data_thingspeak=webread('https://api.thingspeak.com/channels/1016041/feeds.json?api_key=556BHJ6KBKSUA7DV&')