Using API to download data from the Statistics Sweden database (works) and from the NIER database (works not, error message)

This script works for downolading from the Statistics Sweden database,

http://www.statistikdatabasen.scb.se/pxweb/en/ssd/START__NR__NR0103__NR0103C/SektorENS2010KvKeyIn/

%% SCB

url = 'https://api.scb.se/OV0104/v1/doris/en/ssd/START/NR/NR0103/NR0103C/SektorENS2010KvKeyIn';
opts = weboptions('ContentType','auto');
% Whole table
jsonQuery1 = '{"query":[],"response":{"format": "csv"}}';
response1 = webwrite(url, jsonQuery1, opts); % Output is Table in Matlab
Data1 = response1{:,3:end};

Here, response1 is an 18x173 table, and Data1 is an 18x173 matrix. This is perfect for my needs.

However, suppose I try to apply this to the database of the National Institute of Economic Research, for example this table,

http://prognos.konj.se/PxWeb/pxweb/en/SenastePrognosen/SenastePrognosen__f12_hushallensdisponiblainkomster/F1201.px/table/tableViewLayout1/ ,

with the url for the API ,

http://prognos.konj.se/PxWeb/api/v1/en/SenastePrognosen/f12_hushallensdisponiblainkomster/F1201.px

Thus, suppose I apply this:

url = 'http://prognos.konj.se/PxWeb/api/v1/en/SenastePrognosen/f12_hushallensdisponiblainkomster/F1201.px';
opts = weboptions('ContentType','auto');
% {
%   "query": [],
%   "response": {
%     "format": "px"
%   }
% }
jsonQuery = '{"query":[],"response":{"format":"csv"}}';
response = webwrite(url,jsonQuery,opts);

Then I get the following error message:

Error using readContentFromFile

The content reader issued the error message: "'.px' is not a recognized file extension. Unable to detect file type. To read the file as a specific file type, regardless of file extension, use the 'FileType' name-value pair.", while reading the content downloaded from URL

'http://prognos.konj.se/PxWeb/api/v1/en/SenastePrognosen/f12_hushallensdisponiblainkomster/F1201.px'. Use the function WEBSAVE to save the content to a file.

Error in readContentFromWebService (line 66)

readContentFromFile(filename, charSet, urlContentType, options, url);

Error in webwrite (line 139)

[varargout{1:nargout}] = readContentFromWebService(connection, options);

Can you help me solve this and make "response" be a Matlab table?

Answer 1

[Lars Svensson]

Dear Amit,

Thank you very much for this solution and the links.

An alternative solution I just found is the following:

% Download from NIER, set "format" to "csv"
url = ['http://prognos.konj.se/PxWeb/api/v1/en/' ...
    'SenastePrognosen/f12_hushallensdisponiblainkomster/F1201.px'];
opts = weboptions('ContentType','text');
query = '{"query":[],"response":{"format":"csv"}}';
Response = webwrite(url,query,opts);

% Save Response as a text file: response.csv
% Following https://www.mathworks.com/matlabcentral/answers/506670-how-to-create-a-txt-file-from-char-vector-using-matlab
filename = 'Response.csv';  
fid = fopen(filename,'w');     % Open file for writing (overwrite if necessary)
fprintf(fid,Response);         % Write the char array
fclose(fid);                   % Close the file 

% Finally use readtable to create the table
T = readtable(filename,'ReadVariableNames',true,'VariableNamingRule','preserve');

The resulting table T equals the full table on the NIER website.

A question: Is it possible to import Response directly as a table, without first saving it as a file to disk and using readtable?

Answer 2

[Amit Dhakite]

Hi Lars Svensson,

As per my understanding, you are not able to extract the data from NIER Database, as the data present there is in .px format. When I was trying to extract that data mentioning .px format in the query, I was getting an array of dimension [22750 x 1 unit8], which doesn’t meet the requirements.

url = 'http://prognos.konj.se/PxWeb/api/v1/en/SenastePrognosen/f12_hushallensdisponiblainkomster/F1201.px';
query = '{"query": [],"response": {"format": "px"}}';
data = webwrite(url, query);

The data can be obtained in a struct using below code:

url = 'http://prognos.konj.se/PxWeb/api/v1/en/SenastePrognosen/f12_hushallensdisponiblainkomster/F1201.px';
% Mention the format as "json"
query = '{"query": [],"response": {"format": "json"}}';
data = webwrite(url, query);
t = data.data;

You can access the data and create a MATLAB table according to your requirements using that struct. An example for accessing the data is attached below:

Index = ["Current_prices_SEK_million";"Constant_prices_percentage_change";"Current_prices_SEK_million"];
% The values are present in the gap of 40 indices.
Year_1993 = [t(1).values; t(41).values; t(81).values];
Year_1994 = [t(2).values; t(42).values; t(82).values];
Year_1995 = [t(3).values; t(43).values; t(83).values];
RequiredTable = table(Index, Year_1993, Year_1994, Year_1995)

RequiredTable = 3×4 table
                   Index                    Year_1993       Year_1994       Year_1995  
    ___________________________________    ____________    ____________    ____________

    "Current_prices_SEK_million"           {'788990.0'}    {'822728.0'}    {'883842.0'}
    "Constant_prices_percentage_change"    {'NA'      }    {'1.6'     }    {'4.3'     }
    "Current_prices_SEK_million"           {'646219.0'}    {'672630.0'}    {'705651.0'}

The table present on the database:

For more information, kindly go through the following documents:

• How to create a MATLAB Table: Link
• Using webwrite: Link