Loading in raw image data using the Tiff class: Images return as all zeros, multiple warnings

I have image data in the Nikon .NEF raw format that I would like to open in MATLAB for some image processing. To accomplish this, I have followed Steve Eddins' blog post on the subject. Specifically, I am converting my .NEF files to digital negative (.dng) files using the Adobe DNG Converter, then using the Tiff class to read in the data.

I should also note that I am able to view my converted .dng files externally to MATLAB, and there are no apparent problems with the conversion.

My images are 4288x2848 px, and simply calling the imread() function on them returns a smaller thumbnail, which seems consistent with the blog post. Since I would like to view them in full resolution, I run the following lines, copied from the blog:

warning off MATLAB:tifflib:TIFFReadDirectory:libraryWarning
t = Tiff(fullpath,'r'); %fullpath = path of .dng file
offsets = getTag(t,'SubIFD');
setSubDirectory(t,offsets(1));
cfa = read(t);
close(t);

However, the data read into cfa returns as a 2868x4310 uint16 of zeros. I suspect this has something to do with my offsets, but I don't know enough about .tiff and SubIFDs in particular to really know where to go from here.

A few relevant warnings that appear in the code are copy+pasted below:

The following appears after calling Tiff() and getTag():

Warning: TIFF library warning: TIFFReadDirectory: [full path of .dng redacted]: unknown field with tag 37393 (0x9211) encountered..

This warning appears numerous times with numerous tags listed.

The following appears after attempting to read in the data into cfa:

Warning: TIFF library warning: JPEGPreDecode: Improper JPEG strip/tile size, expected 256x240, got 128x240.. 
Warning: TIFF library warning: JPEGLib: Must downscale data from 16 bits to 8..

Apologies if it sounds like I am just dumping this on the forum and saying "fix this," as that is not my intention. As I said, I know next to nothing about .tiff files, so if my problem is one that could be solved simply by reading up on them, relevant sources in lieu of a line-by-line solution would be appreciated.

[Evan]

Addition: I played around a bit with this tiff object, and I noticed that if I set the subIFD to the second value of offset, I get a 680x1024 uint8 that looks to be a scaled-down version of the full resolution image. Again, no idea what's going on here, but possibly it will be of help.

[Evan]

Also, when I call "info" for each subdirectory, the first has a listed size that matches the full resolution, but as you can see, when I read in that subdirectory, I get a larger size. The size of the second subdirectory, however, matches that of the first.

[KAE]

Just to note: I have the same problem (all zeros for the following code), and the DNG file is generated directly by the camera. I don't use Adobe's DNG converter.

warning off MATLAB:tifflib:TIFFReadDirectory:libraryWarning 
t = Tiff(fileImage, 'r'); % Access a DNG image file
rawData = t.read(); 
sum(rawData(:)) % All values are zero

[TaylorB]

I'm having a similar problem working with Nikon raw (NEF) files, having tried all the recommended settings for DNG conversion using the Adobe DNG Converter (Custom settings, unchecked linear, checked uncompressed, and reverse-compatible to DNG 1.3), as well as several variations (with or without JPEG preview, and with or without embedding the original raw file). I have also tried converting the file to DNG using Adobe Lightroom 5.

In addition to Steve Eddin's blog post linked above, I'm following the detailed DNG workflow in the RAW Guide by RC Sumner (http://www.rcsumner.net/raw_guide/). It appears that the DNG conversion doesn't allow access to the information in the SubIFD (or SubIFDs) field (for example, to crop the image to its effective pixels or to access the Linearization Table). Any attempts to access the SubIFD generate many warnings and errors along the way. However, I can at least read the raw sensor data and display the image using imtool.

Can anyone help me to access information from the SubIFD for Nikon NEF files converted to DNG?

The first indication of trouble is when I use the command suggested in Steve's blog:

info = imfinfo('myfile.dng')

which generates a pared-down version of the info structure, with only 19 fields instead of the 62 fields in Steve's example.

I then created the Tiff using the series of commands from RC Sumner's raw guide (similar as in Steve's blog, except first creating the variable filename):

filename = 'myfile.dng'; % Put file name here
warning off MATLAB:tifflib:TIFFReadDirectory:libraryWarning
t = Tiff(filename,'r');

This yields thirteen TIFF Library warnings of similar form as Evan's:

Warning: TIFF library warning: TIFFReadDirectory: unknown field with tag 37393 (0x9211) encountered..

The next series of commands from Section 4.1 of RC Sumner's Raw Guide is supposed to retrieve offset tags from the subIFD and read the raw sensor data from the TIFF object:

offsets = getTag(t,'SubIFD');
setSubDirectory(t,offsets(1));
raw = read(t);
close(t);

However, any attempts fail to extract information from the SubIFDs field (e.g., to crop the image to the effective pixel dimensions or obtain the Linearization curve), e.g., continuing the workflow from Sec. 4.1 of RC Sumner's Raw guide:

meta_info = imfinfo(filename);
% Crop to only valid pixels
x_origin = meta_info.SubIFDs{1}.ActiveArea(2)+1;
width = meta_info.SubIFDs{1}.DefaultCropSize(1);
y_origin = meta_info.SubIFDs{1}.ActiveArea(1)+1;
height = meta_info.SubIFDs{1}.DefaultCropSize(2);
raw = double(raw(y_origin:y_origin+height-1,x_origin:x_origin+width-1)); 

returning the error:

Reference to non-existent field 'SubIFDs'.

Steve's blog uses these commands to access the SubIFD data:

info = imfinfo('myfile.dng')
info.SubIFDs{1}

However, the second command also produces the same error message:

Reference to non-existent field 'SubIFDs'.

[Walter Roberson]

by the way 37393 is "image number" https://www.loc.gov/preservation/digital/formats/content/tiff_tags.shtml

[TaylorB]

Thanks for that useful list--it's a good resource and I'll check it against the other warnings I received.

I'm not sure what I can do about the difficulty of accessing the SubIFD information, which these NEF files seem to have stubbornly hidden. One thought which occurred to me is that the camera might be set to store "compressed raw" images, an option on the D800E. If so, I suppose this might override the custom setting of checking "uncompressed" in the Adobe DNG Converter settings. Unfortunately, I can't check the camera setting now because it's in the lab and there's no access during the virus shutdown.

I downloaded a tool called "Tiff Tag Reader" by Dirk-Jan Kroon (https://uk.mathworks.com/matlabcentral/fileexchange/36958-tiff-tag-reader) that allowed me to see that there are five SubIFD structures in the converted DNG file, but inspection of these structures doesn't yield any linearization table. I'm very new to this, so I'm still exploring what the best way is to handle NEF files in Matlab. I might just use dcraw outside of matlab to process the images, but I was hoping I could implement a workflow entirely within Matlab.

Answer 1

[Evan]

It looks as if my problems stemmed not from the Matlab code, but from Adobe's DNG converter. The below settings in the DNG converter solved my problem:

Preferences > Compatability > Custom > Backward Version > DNG 1.3
Preferences > Compatability > Custom > Uncompressed

[KAE]

These settings worked for me except the backward version I used was "DNG 1.6". After that Matlab had no problems reading in the DNG as non-zero values.

Answer 2

[Reem]

I have tried the same Adobe DNG Converter settings .. and whenever I tired

 imtool(cfa)

to display my image I got a black image. Any clue?!

Answer 3

[Image Analyst]

One option that worked for me is to use a very inexpensive image management program, ACDSee, http://www.acdsee.com/ to convert the file to a standard .PNG format. You can select one of more image files in ACDSee by clicking on their thumbnails and then simply right click on the .NEF file and select Batch->ConvertFileFormat and select .PNG, .TIFF, or any of a bunch of other image formats. Then you just read in a standard image format file, like PNG, with MATLAB.

Answer 4

[Image Analyst]

Try responding to Steve by posting a link to this question on his blog. Don't post your whole question, just say you have a question on http://www.mathworks.com/matlabcentral/answers/81448-loading-in-raw-image-data-using-the-tiff-class-images-return-as-all-zeros-multiple-warnings and ask if he can help.

[Evan]

Thanks. I'll give it a try.

[Evan]

Actually, it looks as if comments have been closed for that particular entry.

[Image Analyst]

Call the Mathworks and ask for Ashish Uthama - he works on image formats.

[Noor Abbas]

Hi,

Does anyone get answer how to read raw image .nef in matlab?

Please help