Multiple File Input Processing

Hello, I need to process multiple input files in a loop and need some assistance. Here is my problem

I'm currently tasked to evaluate satellite rainfall data files (format *.grb, contains a rainfall data matrix for each geo-pixel observed by satellite - about 3000x3000). I have thousands of input files been taken for every 30 minutes interval. Here below is some example file names for one of those input files:

MSG2-SEVI-MSGMPEG-0100-0100-20070605003000.000000000Z-999804.grb

MSG2-SEVI-MSGMPEG-0100-0100-20070605010000.000000000Z-999804.grb

MSG2-SEVI-MSGMPEG-0100-0100-20070605013000.000000000Z-999804.grb

Considering first one as an example, the numbers in the middle indicates the date and time: 2007/06/05 - 00:30. And to read these kind of files (*.grb), I have a code called ReadGRIB2.m (.grb files is not directly readable on Matlab)

Rather than trying to make filenames recognized by Matlab, I just typed dir('*.grb') so matlab can read the grb files on same directory. Here is a reduced overview of my code:

Filename = dir('*.grb');
for i=1:length(Filename)
eval(['load' Filename(i).name ' .grb']);
% Decoding of *.grb file
[Data,Info]=readGRIB2(Filename);

% Presentation of Data - Creates output data wrt input filename
msg2projection(Data,Lat,Lon);
end

When I run the code, I get the following error:

??? Error using ==> load
Number of columns on line 2 of ASCII file
...\MSG2-SEVI-MSGMPEG-0100-0100-20070605003000.000000000Z-999804.grb
must be the same as previous lines.

Error in ==> MPE at 21
eval(['load ' Filename(i).name ' .grb']);

What I understand from error: I think Matlab is unable to mount the file into the ReadGRIB2 function because of being unable to recognize it first. I don't have much experience on Matlab and I would be very faithful if you can assist me.

Thanks very much in advance.

Answer 1

[Oleg Komarov]

First of all you don't need to use eval for this task.

load(Filename(i).name);

Second, does the data file contain headers? How is it exactly structured the file?

EDIT I found this link: http://oiswww.eumetsat.int/~idds/html/grib.html

Grib files are binary, so you cannot use high level import functions but you should decode them.

To import GRIB data:

• http://www.renci.org/~bblanton/files/ReadGrib/read_grib.html
• http://www.mathworks.com/matlabcentral/fileexchange/21579

In brief just erase the line with the eval from your code and if readGRIB2 is a valid decoding routine you should be fine.

[Al Onen]

Files are not directly readable on Matlab as I wrote, so even with the readGRIB2 function, it comes out just as a matrix. When I open the input files with notepad, the below lines appear and the rest of the file is encoded gibberish

Example 1st File:

êu

2 !Cb

HeaderVersionNo 0

FileType 2

SubHeaderType 1

SourceFacilityID 6

SourceEnvId 1

SourceInstanceId 2

SourceSUId 60021

SourceCPUId 10 20 50 4

DestFaciliyId 5

DestEnvId 1

DataFieldLength 2179938

«F„ *iµB

SubHeaderVersionNo 0

ServiceType 171

ServiceSubType 24

FileTime 18052: 2779573

SpacecraftId 322

Example 2nd File:

êu

2 !I

HeaderVersionNo 0

FileType 2

SubHeaderType 1

SourceFacilityID 6

SourceEnvId 1

SourceInstanceId 2

SourceSUId 60021

SourceCPUId 10 20 50 4

DestFaciliyId 5

DestEnvId 1

DataFieldLength 2166857

«F„ EÜ“B

SubHeaderVersionNo 0

ServiceType 171

ServiceSubType 24

FileTime 18052: 4578451

SpacecraftId 322

[Oleg Komarov]

The link I provided and the readGRIB2 function are supposed to import the data decoding it.

[Al Onen]

I dont have a problem with readGRIB2 function when I'm trying to import one file, so does "Grib files are binary, so you cannot use high level import functions but you should decode them." mean I cannot use a loop to process all of them in a single attempt?

[Oleg Komarov]

A loop by definition is not a single attempt. You can use a loop to process them all with readGRIB2, at this point I don't understand what's your problem.

[Al Onen]

Shortly, the for loop is just not working while i.e. readGRIB2(MSG2-SEVI-MSGMPEG-0100-0100-20070605003000.000000000Z-999804.grb) works without a problem. I did it like you said, but still failed.

Thanks for your assistance by the way, I really appreciate your patience. I do apologize for the lack of expressing my problem :).

[Oleg Komarov]

The for loop is just not working says nothing. Did you get an error or an unexpected result?

[Al Onen]

I get the same error on the very first post.

[Oleg Komarov]

Why are you still using load? Use readGRIB2 directly.

[Al Onen]

Ah, I now got it. It works as I wanted. Thank you so much Oleg.

[Oleg Komarov]

If my answers were useful please accept it.