dlm read error in data import - alternatives?

I am looking for a workaround to an error I am getting in an m-file. It is from the Matlab Cenral File exchange: http://www.mathworks.com/matlabcentral/fileexchange/25764-windcan

The m-file downloads climate data from a website and names it, then appends only the data the user needs its own file.

      function windcan
      % COPYRIGHT (C) 2009, JOSH WIEBE
      % DISTRIBUTED "AS IS" UNDER THE BSD SOFTWARE LICENCE, SEE    LICENCE FILE
      % THIS PROGRAM USES THE URLWRITE COMMAND TO DOWNLOAD HOURLY CLIMATE DATA FROM THE ENVIRONMENT CANADA WEBSITE
      % WIND DATA IS THEN PLUCKED FROM THE DOWNLOADED FILES AND SAVED IN A CSV FILE WHICH CAN LATER BE ANALYSED
      % IN WRPLOT OR A SIMILAR PROGRAM, OUTPUT FILE COLUMN ORDER IS YEAR, MONTH, DAY, HOUR, DIRECTION, SPEED

      fprintf ('See http://climate.weatheroffice.ec.gc.ca/climateData/canada_e.html \n select location, scroll to link to "bulk data", station id appears in menu bar. \n e.g. Ottawa = 4337 \n');
      stationid = input('Enter Station ID  ');
      year1 = input('Enter start year  ');   
      year2 = input('Enter end year  '); 
      month1 = input('Enter start month, enter 1 for January '); 
      month2 = input('Enter end month  '); 
      filename = input('Enter output file name:  ', 's'); 

      N = (month2-month1+1)*(year2-year1+1);
      year = year1;
      month = month1;

      % ROUTINE TO DOWNLOAD CLIMATE DATA
      for i=1:N
      urlwrite(sprintf('http://www.climate.weatheroffice.ec.gc.ca/climateData/bulkdata_e.html?timeframe=1&Prov=XX&StationID=%i&Year=%i&Month=%i&Day=1&format=csv&type=hly', stationid, year, month), sprintf('%i%i%i.csv', stationid, year, month));
        if month < month2;
          month = month +1;
        else
          month = month1;
          year = year+1;
        end %if
      end %for

      % RESET PARAMETERS, ASSIGN NAME FOR OUTPUT FILE
      year = year1;
      month = month1;
      s = 'YEAR, MONTH, DAY, HOUR, DIRECTION, SPEED (m/s)';
      dlmwrite(filename, s, '');

      % ROUTINE TO PLUCK WIND DATA FROM DOWNLOADED CLIMATE DATA
      for i=1:N
        temp=dlmread(sprintf('%i%i%i.csv', stationid, year, month), ' "," ');  % ' "," ' USED AS DELIMITER BECAUSE THE .CSV FILE USES QUOTATION MARKS AROUND TEXT FIELDS
        L=length(temp);
        yearI = temp (15:L, 6); % START IMPORT AT ROW 15, COLUMN 6 IS YEAR
        monthI = temp (15:L, 9);
        dayI = temp (15:L, 12);
        hourI = temp (15:L, 15);
        winddir = temp (15:L, 36);
        windspd = temp (15:L, 42);
        winddat(:,1) = yearI; % PUT DATA INTO ONE MATRIX
        winddat(:,2) = monthI;
        winddat(:,3) = dayI;
        winddat(:,4) = hourI;
        winddat(:,5) = winddir*10; % WIND DIRECTION IN 10 DEGREE BINS
        winddat(:,6) = windspd*0.278; % CONVERT WINDSPEED FROM KPH TO MPS
        dlmwrite(filename, winddat, ',' , '-append'); %APPEND MATRIX "WINDDAT" TO OUTPUT FILE
        clear winddat; %CLEAR THE WINDDAT VARIABLE ONLY

        if month < month2;
          month = month +1;
        else
          month = month1;
          year = year+1;
        end %if
      end %for
      end %function

The code for downloading and naming the files works well - the files are downloaded and name correctly and the new file is created. The last section for retrieving the columns to write to the new file gives this error:

??? Error using ==> dlmread at 75 DELIMITER must be a single character.

Error in ==> windcan at 39 temp=dlmread(sprintf('%i%i%i.csv', stationid, year, month), ' "," '); % ' "," ' USED AS DELIMITER BECAUSE THE .CSV FILE USES QUOTATION MARKS AROUND TEXT FIELDS

First off, why would the author think that ' "," ' would work as a delimiter with dlmread to trick it to importing dates? Can this be done successfully?

Secondly, is there an replacement for dlmread for reading in the file columns without changing the whole naming and looping system? i.e. going to a system that creates a date vector out of the file names and incrementing rather than using the i=1:N file system.

I like the simplicity of the code and the naming system and would like to stay with it if possible. Any help would be greatly appreciated.

Answer 1

[Matt Tearle]

Give this a try

function windcan

fprintf ('See http://climate.weatheroffice.ec.gc.ca/climateData/canada_e.html \n select location, scroll to link to "bulk data", station id appears in menu bar. \n e.g. Ottawa = 4337 \n');
stationid = input('Enter Station ID  ');
year1 = input('Enter start year  ');
year2 = input('Enter end year  ');
month1 = input('Enter start month, enter 1 for January ');
month2 = input('Enter end month  ');
filename = input('Enter output file name:  ', 's');
N = (month2-month1+1)*(year2-year1+1);
year = year1;
month = month1;
% ROUTINE TO DOWNLOAD CLIMATE DATA
for i=1:N
    urlwrite(sprintf('http://www.climate.weatheroffice.ec.gc.ca/climateData/bulkdata_e.html?timeframe=1&Prov=XX&StationID=%i&Year=%i&Month=%i&Day=1&format=csv&type=hly', stationid, year, month), sprintf('%i%i%i.csv', stationid, year, month));
    if month < month2;
        month = month +1;
    else
        month = month1;
        year = year+1;
    end %if
end %for
% RESET PARAMETERS, ASSIGN NAME FOR OUTPUT FILE
year = year1;
month = month1;
s = 'YEAR, MONTH, DAY, HOUR, DIRECTION, SPEED (m/s)';
dlmwrite(filename, s, '');
% ROUTINE TO PLUCK WIND DATA FROM DOWNLOADED CLIMATE DATA
for i=1:N
    fid = fopen(sprintf('%i%i%i.csv', stationid, year, month),'rt');
    data = textscan(fid,['%q',repmat('%*q',1,11),'%q%*q%q',repmat('%*q',1,10)],...
        'headerlines',17,'delimiter',',');
    fclose(fid);

      dv = datevec(data{1});
      winddat = [dv(:,1:4),10*str2double(data{2}),0.278*str2double(data{3})];
      dlmwrite(filename, winddat, '-append', 'delimiter', ',');
      clear winddat; %CLEAR THE WINDDAT VARIABLE ONLY
      if month < month2;
          month = month +1;
      else
          month = month1;
          year = year+1;
      end %if
  end %for
end %function

The format of the CSV files seems to have changed, as has the functionality of the various import functions. This uses textscan to read quoted strings (and convert to numeric).

[Braden]

Thanks for the pointers Matt! I really appreciate the help. This has definitely solved my problem.

The format of the EC climate data logs has definitely not changed in the last 5 months...so I did not suspect that that was the issue. Thanks again.

Answer 2

[Oleg Komarov]

I suggest to replace the dlmread with texscan jumping the headerlines.

Also, review the format of the file since it seems to me that the submission is not up-to-date.

EDIT

Execute this bunch of commands:

stationid = 4337;
year1 = input('Enter start year  ');   
year2 = input('Enter end year  '); 
month1 = input('Enter start month, enter 1 for January '); 
month2 = input('Enter end month  '); 

N = (month2-month1+1)*(year2-year1+1);
year = year1;
month = month1;
i = 1;
% ROUTINE TO DOWNLOAD CLIMATE DATA
urlwrite(sprintf(['http://www.climate.weatheroffice.ec.gc.ca/climateData...
                 /bulkdata_e.html?timeframe=1&Prov=XX&StationID=%i&Year=%i...
                 &Month=%i&Day=1&format=csv&type=hly', stationid,year,month])...
                 ,sprintf('%i%i%i.csv', stationid, year, month));

It will create a csv file on your current directory. Open the csv file and check that the data actually starts from line 18 and there are no more than 30 columns.

[Braden]

Review the format of which file, the climate data log?

What do you mean, the submission is not up-to-date?

Please clarify, thanks!