Ocean Color Forum - Not logged in
Hi,
How are missing values in the Aqua-MODIS level-3 chlorophyll and SST data extracted by SEADAS labelled? i.e. what values are they marked with. This information is necessary for further processing during analysis. Thanks for any help you may provide.
By @long
Date 2007-04-11 00:15
Default missing values used in MODIS Aqua
L2 chlor_a: -1
sst: -32767
L3
chlor_a: 0
sst: 0
You can check them out with 'cursor position' function on
a displayed image.
Long
Thank you Long, 'cursor position' was helpful, although the missing values are showing as 255 for this L3 chlor_a data. Is this right, and is it also true for the output ascii data?
By @long
Date 2007-04-11 12:38
Edited 2007-04-11 12:51
Yes, L3 SMI images showed chlor_a missing values as 255.
L3 binned files are 0.
Long
Thanks again Long. Being a newbie in ocean color, I found that 'cursor position' also reported the raw and geophysical data (which is what I'm getting via ascii output). The 255 raw value for NaN corresponds to 66.83 geophys data.
Sorry for yet another question. What is the difference between L3 SMI images and L3 binned files? I see that the SMI files I obtained from Seawifs contain some values that are not valid, such as > 10 mg/m3 in the Southern Ocean. Is this corrected for in the binned files and if so, how can I obtain these? Thanks for your help.
By mike
Date 2007-04-11 17:33
SMI files are actually L3 binned files projected to a pre-defined map. Therefore L3 binned files precede the L3 SMI files. You might want to read more about these data formats
here.
As far as values > 10 mg/m3 in the Southern Ocean, my guess is that this data passed all the flag/mask criteria and therefore may in fact be valid or else require evaluation/intervention from the end user. When running smigen there are min/max defaults set for each product, but the user can also override these defaults.
And you can produce binned files yourself using SeaDAS or you can access them on our ftp server:
ftp://oceans.gsfc.nasa.gov/SeaWiFS/Binned/ftp://oceans.gsfc.nasa.gov/SeaWiFS/L3BIN/After you've read the format specs if you still have questions, please ask..
Thank you for these pointers. I'm learning about each format specs, and looking at the binned files in the ftp site I'm having trouble interpreting some of the files. For the monthly products I'm interested in, I understand I need to get the main files plus the product file I'm interested in (suffix x07 in my case, for chl_a). However, I see there are other files suffixed 'LND.main' 'PAR.main' and 'PAR.x00', which I can't see a description for in the format specs documents.
I tried one sample main and chl_a product and cannot find how to output the ascii unprojected data (i.e. just in geographic coordinates). The display menu shows several projections, without a geographic option. How can I output the data in unprojected geographic coordinates? Thanks.
By @long
Date 2007-04-11 23:23
Edited 2007-04-12 00:03
L3 binned HDF files save SDSs in HDF vgroup. You can access those
data values with IDL commands. You may refer my script to
write ascii output directly from those L3 HDF files.
Long
PRO READ_TEST
fname='M20021852002210.L3b_GAC.L3b_GAC'
fid = hdf_open(fname)
chl=hdf_vd_find(fid,'chlor_a')
cid = hdf_vd_getid(fid,chl)
e=hdf_vd_attach(fid,cid)
hdf_vd_get,e,name=n,size=s,count=co
nrec=hdf_vd_read(e,chlor_a)
print,'chlor_a array: ', size(chlor_a)
print,'sample data = '
print, chlor_a(0,0:10)
; openw,1,'out.asc'
; printf,1,chlor_a
; close,1
sid=hdf_sd_start(fname)
v=hdf_sd_attrfind(sid,'Northernmost Latitude')
hdf_sd_attrinfo,sid,v,data=lat1
print,'Northernmost Latitude : ', lat1
hdf_sd_end, sid
. . . . .
END
Screen dump messages are:
SeaDAS> read_test
chlor_a array: 2 2 3383483 4 6766966
sample data =
29.2700
41.4329
50.8934
41.4647
29.2350
50.6740
41.3905
42.0481
41.9739
51.2543
29.3800
Northernmost Latitude : 41.1406
Dear Long,
I'm completely inexperienced with IDL, so couldn't follow your script. However, I found a simpler script that seems to extract the data, albeit not at the full resolution in the file. Particularly, I don't understand what the if statement does in the for loop, and perhaps that needs to be changed to work with the L3 monthly binned chlorophyll data.
pt1=[170, 60] & pt2=[180, 80]
wdir='/some-dir/*.main'
ff=findfile(wdir)
n=n_elements(ff)
for i=0, n-1 do begin & $
pos=strpos(ff, '/', /reverse_search) & $
sensor=strmid(ff, pos+1, 1) & $
seadisp & $
ascii2=ff + '_geo.asc' & $
if (sensor eq 'A') then ftype='MODIS' else ftype='SeaWifs' & $
load, ff, ftype=ftype, prod_name='chlor_a' & $
out_ascii, ascii2, band_no=1, region=[pt1, pt2], vars=[4, 3, 2], $
format='(F10.2, F10.2, F10.4)', $
titles=['lon', 'lat', 'chl_a'], /ll_flag & $
clear_up & $
endfor
I'd greatly appreciate any help to get this to output at the full resolution of the L3 binned files. Thanks once again.
By @long
Date 2007-04-12 02:49
This script is used for loading, displaying and output ASCII text from
defined regions. The For loop is good for multiple SeaWiFS/MODIS L3 files.
That IF statement separates MODIS files from SeaWiFS files.
Those L3 files will be load/displayed with default projection 'cylindrical'.
As a new user, you should start with SeaDAS's GUI mode + single file.
To realize ascii data output (similar to that script's work), steps are:
(1) load L3 file with a projection selection
(2) functions -> output -> data -> Ascii -> Setup (format) ->Write File
My script was created for direct read out from L3 files because you
mentioned 'output data in unprojected geographic coordinates'.
Long
So if I understand correctly, I cannot output the ascii data in unprojected geographic coordinates with the GUI and would have to use a script as you showed. I can always unproject the data elsewhere, so this is not too serious. However, in the GUI I see that I'm given the option of 'output size', and varying this changes the resolution of the output file. This has got me confused, since this is L3 monthly binned data so it can only have one possible resolution. Am I missing something?
I adapted your script to my situation, and it returns an error triggered by the line with the nrec variable:
SeaDAS Version 5.0.5 (pid = 12985)
% HDF_VD_READ: Unable to read VDATA (1073741824).
[MAIN]: Error executing the command: nrec=hdf_vd_read(e, chlor_a)
MAIN: Error executing command in batch file
Any idea what may be wrong?
By @long
Date 2007-04-12 14:54
I used L3 spatial/temporal binned file as input
in my script. Make sure the product name is
'chlor_a' (display image and check).
Script will not work for L3 SMI files (additional
mapping process involved).
Long
By @long
Date 2007-04-12 15:17
SeaDAS uses interpolation/subsampling to create various sized images
for display and output at L3. L2 files have original resolution.
Long
I'm also using an L3 binned file (S20012442001273.L3b_MO.main), not an SMI file, and the product name is indeed 'chlor_a'. This is my adapted script (only trying up to the failing line; the rest is commented):
ff=hdf_open('S20012442001273.L3b_MO.main')
chl=hdf_vd_find(ff, 'chlor_a')
cid=hdf_vd_getid(ff, chl)
e=hdf_vd_attach(ff, cid)
hdf_vd_get, e, name=n, size=s, count=co
nrec=hdf_vd_read(e, chlor_a)
Of course, I have the *.x07 chlorophyll file in the same directory as the *.main file.
Is it absolutely not possible to do this from the GUI?
By @long
Date 2007-04-12 18:11
Edited 2007-04-12 18:21
Thank you Long, I can reproduce this now. However, I don't understand why this doesn't work with my monthly binned data. What do you mean by putting the chlorophyll file as first? I only have the chlorophyll file for each monthly main file (and I have a couple of dozen of them in the same directory). How should I modify the script to work in this case? Thanks.
By @long
Date 2007-04-12 21:13
Edited 2007-04-12 21:27
If you only have 'chlor_a', don't worry about the file order. Otherwise,
the script will first read .x07 then .x08. My files .x07 is chlor_a
and .x08 is K_490.
To process multiple files, you should consider use For loop:
l3file=findfile('S*.main')
n=n_elements(l3file)
for i=0, n-1 do begin
fname=strmid(l3file(i),0,19)
ff=hdf_open(l3file(i))
chl=hdf_vd_find(ff, 'chlor_a')
cid=hdf_vd_getid(ff, chl)
e=hdf_vd_attach(ff, cid)
hdf_vd_get, e, name=n, size=s, count=co
nrec=hdf_vd_read(e, chlor_a)
outfile = fname + '.asc'
openw, 1, outfile
printf, 1, chlor_a
close,1
clear_up
endfor
Yes, eventually I would have to use some loop to output all the monthly files. Thanks for showing how this is done in idl. What command should I use to append the coordinates for each data value also?
By @long
Date 2007-04-12 22:03
Edited 2007-04-12 22:09
What command should I use to append the coordinates for each data value also?
You are coming back to projections. The most reliable and convenient way in using L3 data
with coordinate systems is direct use SeaDAS Display functions.
Display an image and select a projection then output data in various forms (binary, ascii,
graphic, etc.). Users can transfer projections from one to another within SeaDAS's image
display functions.
Try to directly use SeaDAS for your L3 data, other ways may work but
lack of confidence for the reliability.
Long
I must be misunderstanding something. I need to obtain the chlorophyll values with the associated unprojected geographic coordinates. The GUI does not offer an option to output the data this way, so I can output the data but I must choose a projection and a size (performing the interpolation/subsampling as you mentioned).
Using your script for the output of chlorophyll data, how do I know to which lon and lat they correspond to?
By @long
Date 2007-04-13 04:00
L3 files have regular grid. With known array size, northernmost,
southernmost, easternmost and westernmost positions. You can
estimate each pixel/line's lat/lon.
Wish this is what you want. You need check those geo-info carefully
after your work.
If you use SeaDAS's ascii output, you can have formatted lat/lon/data_value
easily.
Long
Ok, I think I made some painful progress with this. Adapting your scripts to my case, the following is very close to what I need:
region=[-southlat, westlon, northlat, eastlon]
projfile="/path/to/projdata"
;; Loop through all the main files
l3file=findfile('S*.main')
n=n_elements(l3file)
for i=0, n-1 do begin & $
fname=strmid(l3file(i), 0, 22) & $
outfile=fname + '.asc' & $
load, l3file(i), prod_name='chlor_a', limit=region, map_pfile=projfile & $
out_ascii, outfile, band_no=1, vars=[4, 3, 2], titles=['lon', 'lat', 'chl_a'] & $
clear_up & $
endfor
where the projdata file was obtained choosing 'save' from the GUI. It outputs the data as I need them. However, I don't understand how I'm supposed to choose an appropriate image size to maintain the resolution in the actual data (i.e. 9 km for Seawifs L3 binned). How can I calculate that? Also, the projdata file doesn't contain full projection info that I can use to unproject it later (datum and ellipsoid missing, although I'm probably worrying too much and WGS84 is close enough). The easiest would probably be to leave the projection at the default cylindrical projection, but I need to know what projection parameters Seadas uses, so I can unproject the data anyway. Any further input appreciated.
By @long
Date 2007-04-13 13:27
You can estimate an image size to match
9 km resolution for your selected region.
L3 data are regular gridded and should be
easy to do.
At 9km resolution, choice of datum will make
no noticeable difference.
The default ellipsoid should be the Clarke 1866 ellipsoid,
[6378206.4, 0.00676866, 0.9996] in IDL mapping programs.
(I need reconfirm this).
Yes, WGS84 should be colse enough.
Long
Do you mean that simply calculating the number of km in x and y in my region and dividing that by 9 should give a matching image size? i.e.:
(degrees * (cos(projected center lat)*111.325)) / 9
Could you please confirm that the output ascii data (from the GUI's output function) are in the projection selected during loading? Thanks for your help.
By @long
Date 2007-04-13 18:28
Yes, your estimation seems right to me.
And you are right that SeaDAS created ascii output will
associate with selected map projection.
Long
By mike
Date 2007-04-13 18:37
I think maybe you want to backtrack a little here.. it sounds like directly accessing the bins is what you really want, so it's best to avoid all the unneeded projection headaches. I'm hoping the following post will help you to do exactly what you need:
http://oceancolor.gsfc.nasa.gov/forum/oceancolor/topic_show.pl?pid=3604#pid3604I realise SeaDAS should have a feature to extract binned data values and lat/lons and it is high on our list of things to do.. hopefully delving into all of this will be of some use to you in the long run!
By mike
Date 2007-04-13 18:41
Also, since you already have SeaDAS, the HDF4 distribution required to build swreadl3b is in your $SEADAS/hdf/HDF4/ directory.
By mike
Date 2007-04-13 18:56
I also created a new
FAQ posting on this subject with a link to the BIN2LL IDL utility as well.
Yes!! That's exactly what I need. I'll follow that path, as the image size estimation and projection/inverse-projection problems are very difficult to solve satisfactorily. Thanks for pointing this out.
By mike
Date 2007-04-13 20:42
I'm not sure what is going on.. your make_log has all kinds of non-ASCII characters in it. Do you see these as well? Is your system's language set to non-english perhaps?
I just grabbed the Makefile and swreadl3b.c files and changed two lines in the Makefile:
> HDFROOT = /Users/mike/seadas5.0/hdf/HDF4
> LIBS = -ldf -ljpeg -lz -lsz -lm
Typing "make" built the swreadl3b binary with no errors.
Can you make the file /home/sluque/.seadas5.0/hdf/HDF4/include/hdf.h available on your website for me to see?
By mike
Date 2007-04-13 20:54
It seems like your compiler is doing something weird.. can you try to compile this simple program called hello.c:
#include <stdio.h>
int main() {
int a=1;
printf("Hello World, %d\n", a);
return 1;
}
gcc -ansi -pedantic -g -o hello hello.c
./hello
Hello World, 1
I don't see any non-ASCII stuff in that make_log, even opening it from http in a browser. I also just did "make" (I put a new make_log
http://members.shaw.ca/sluque, also with the modified Makefile and copy of hdf.h you mentioned). Thanks.
Nothing strange with that though:
gcc -ansi -pedantic -g -o hello hello.c
./hello
Hello World, 1
By mike
Date 2007-04-13 21:19
Here's what the 5th line of your make_log looks like to me:
/home/sluque/.seadas5.0/hdf/HDF4/include/hdfi.h:1267: error: expected âEUR~=âEUR(TM), âEUR~,âEUR(TM), âEUR~;âEUR(TM), âEUR~asmâEUR(TM) or âEUR~__attribute__âEUR(TM) before âEUR~machineâEUR(TM)
...further down in the file the first swreadl3b.c error is:
swreadl3b.c:40: error: expected âEUR~=âEUR(TM), âEUR~,âEUR(TM), âEUR~;âEUR(TM), âEUR~asmâEUR(TM) or âEUR~__attribute__âEUR(TM) before âEUR~basebinâEUR(TM)
...line 40 of swreadl3b.c is simply:
static int32 basebin[NUMROWS];
Can you paste these lines as you see them in make_log? It almost seems like your compiler is having trouble reading the source code, and is therefore seeing it as mangled.
What operating system are you using?
Is your system language set to U.S. English?
What version of SeaDAS do you have installed?
By mike
Date 2007-04-13 21:26
Mike,
I'm pasting below the first 50 lines of my make_log (hopefully it won't
get mangled this way):
sluque@patagonia[seadas]$ make
gcc -ansi -pedantic -g -I/home/sluque/.seadas5.0/hdf/HDF4/include -o swreadl3b swreadl3b.c -I/home/sluque/.seadas5.0/hdf/HDF4/include -L/home/sluque/.seadas5.0/hdf/HDF4/lib -ldf -ljpeg -lz -lsz -lm
In file included from /home/sluque/.seadas5.0/hdf/HDF4/include/hdf.h:19,
from swreadl3b.c:15:
/home/sluque/.seadas5.0/hdf/HDF4/include/hdfi.h:1267: error: expected ?=?, ?,?, ?;?, ?asm? or ?__attribute__? before ?machine?
In file included from swreadl3b.c:15:
/home/sluque/.seadas5.0/hdf/HDF4/include/hdf.h:28: warning: data definition has no type or storage class
In file included from swreadl3b.c:15:
/home/sluque/.seadas5.0/hdf/HDF4/include/hdf.h:133: error: expected ?=?, ?,?, ?;?, ?asm? or ?__attribute__? before ?HFILEID?
/home/sluque/.seadas5.0/hdf/HDF4/include/hdf.h:135: error: expected declaration specifiers or ?...? before ?*? token
/home/sluque/.seadas5.0/hdf/HDF4/include/hdf.h:135: error: ?intn? declared as function returning a function
In file included from /home/sluque/.seadas5.0/hdf/HDF4/include/hdf.h:142,
from swreadl3b.c:15:
/home/sluque/.seadas5.0/hdf/HDF4/include/hbitio.h:35: error: expected specifier-qualifier-list before ?int32?
/home/sluque/.seadas5.0/hdf/HDF4/include/hbitio.h:52: warning: struct has no members
/home/sluque/.seadas5.0/hdf/HDF4/include/hbitio.h:57: error: expected ?=?, ?,?, ?;?, ?asm? or ?__attribute__? before ?maskc?
/home/sluque/.seadas5.0/hdf/HDF4/include/hbitio.h:67: error: expected ?=?, ?,?, ?;?, ?asm? or ?__attribute__? before ?maskl?
In file included from /home/sluque/.seadas5.0/hdf/HDF4/include/hdf.h:143,
from swreadl3b.c:15:
/home/sluque/.seadas5.0/hdf/HDF4/include/hcomp.h:56: error: expected ?=?, ?,?, ?;?, ?asm? or ?__attribute__? before ?compress_map?
/home/sluque/.seadas5.0/hdf/HDF4/include/hcomp.h:80: error: expected specifier-qualifier-list before ?int32?
/home/sluque/.seadas5.0/hdf/HDF4/include/hcomp.h:84: warning: struct has no members
/home/sluque/.seadas5.0/hdf/HDF4/include/hcomp.h:93: error: field ?quality? declared as a function
/home/sluque/.seadas5.0/hdf/HDF4/include/hcomp.h:95: error: field ?force_baseline? declared as a function
/home/sluque/.seadas5.0/hdf/HDF4/include/hcomp.h:105: error: expected specifier-qualifier-list before ?int32?
/home/sluque/.seadas5.0/hdf/HDF4/include/hcomp.h:111: warning: struct has no members
/home/sluque/.seadas5.0/hdf/HDF4/include/hcomp.h:115: error: field ?skp_size? declared as a function
/home/sluque/.seadas5.0/hdf/HDF4/include/hcomp.h:121: error: field ?level? declared as a function
In file included from /home/sluque/.seadas5.0/hdf/HDF4/include/hdf.h:144,
from swreadl3b.c:15:
/home/sluque/.seadas5.0/hdf/HDF4/include/herr.h:104: error: expected ?=?, ?,?, ?;?, ?asm? or ?__attribute__? before ?error_top?
In file included from /home/sluque/.seadas5.0/hdf/HDF4/include/hdf.h:145,
from swreadl3b.c:15:
/home/sluque/.seadas5.0/hdf/HDF4/include/hproto.h:64: error: expected ?=?, ?,?, ?;?, ?asm? or ?__attribute__? before ?Hopen?
/home/sluque/.seadas5.0/hdf/HDF4/include/hproto.h:68: error: expected ?)? before ?file_id?
/home/sluque/.seadas5.0/hdf/HDF4/include/hproto.h:70: error: expected ?=?, ?,?, ?;?, ?asm? or ?__attribute__? before ?Hstartread?
/home/sluque/.seadas5.0/hdf/HDF4/include/hproto.h:74: error: expected ?)? before ?access_id?
/home/sluque/.seadas5.0/hdf/HDF4/include/hproto.h:77: error: expected ?)? before ?file_id?
/home/sluque/.seadas5.0/hdf/HDF4/include/hproto.h:80: error: expected ?)? before ?access_id?
/home/sluque/.seadas5.0/hdf/HDF4/include/hproto.h:84: error: expected ?=?, ?,?, ?;?, ?asm? or ?__attribute__? before ?Hstartwrite?
/home/sluque/.seadas5.0/hdf/HDF4/include/hproto.h:87: error: expected ?=?, ?,?, ?;?, ?asm? or ?__attribute__? before ?Hstartaccess?
/home/sluque/.seadas5.0/hdf/HDF4/include/hproto.h:91: error: expected ?)? before ?file_id?
/home/sluque/.seadas5.0/hdf/HDF4/include/hproto.h:94: error: expected ?)? before ?aid?
/home/sluque/.seadas5.0/hdf/HDF4/include/hproto.h:97: error: expected ?)? before ?aid?
/home/sluque/.seadas5.0/hdf/HDF4/include/hproto.h:100: error: expected ?)? before ?term_func?
/home/sluque/.seadas5.0/hdf/HDF4/include/hproto.h:103: error: expected ?)? before ?access_id?
/home/sluque/.seadas5.0/hdf/HDF4/include/hproto.h:105: error: expected ?=?, ?,?, ?;?, ?asm? or ?__attribute__? before ?Htell?
/home/sluque/.seadas5.0/hdf/HDF4/include/hproto.h:108: error: expected ?=?, ?,?, ?;?, ?asm? or ?__attribute__? before ?Hread?
/home/sluque/.seadas5.0/hdf/HDF4/include/hproto.h:111: error: expected ?=?, ?,?, ?;?, ?asm? or ?__attribute__? before ?Hwrite?
/home/sluque/.seadas5.0/hdf/HDF4/include/hproto.h:114: error: expected ?=?, ?,?, ?;?, ?asm? or ?__attribute__? before ?Htrunc?
and the line with swreadl3b.c:
swreadl3b.c:40: error: expected ?=?, ?,?, ?;?, ?asm? or ?__attribute__? before ?basebin?
I've seen these things happen when a locale issue is involved, but don't
know how to fix it here. My system is a Debian GNU/Linux with kernel:
2.6.18-4-amd64 #1 SMP Mon Mar 26 11:36:53 CEST 2007 x86_64 GNU/Linux
and my system language is indeed english, although I'm on a en_CA.UTF-8
locale. I'm using Seadas version 5.0.5. Were you able to see the hdf.h
file without non-ASCII characters? Maybe that helps pinpoint whether it's
compiler issue, or the locale...
I didn't see any mangled characters in the original Makefile and
swreadl3b.c files, but using your new gzip'd files also gave me the same
errors, so there's something else going on. This is puzzling.
Maybe some other flag is needed for this to work with AMD64 ?
By mike
Date 2007-04-13 22:02
Your hdf.h file looked fine to me.. so yeah I'm puzzled too. I built the file with no problems on Mac OS X, RedHat 7.3, and Fedora Core 4. Since the Fedora Core 4 kernel is the closest one we have to yours, try using the binary I built:
ftp://samoa.gsfc.nasa.gov/seadas/pub/spluque/swreadl3b.gz
I'll try your binary. Does it include the modification for the L3 bin files mentioned in the original post describing this tool?
I'll also try building this inside a 32 bit chroot and report back.
Ok, good news is that your binary works well in my system!
Also, building swreadl3b in a 32-bit chroot went without any problems in a split second, and surprisingly runs well in the AMD64 main system. So hope this helps others in a similar situation. However, I wonder what the problem is here (really sounds like something is needed for it to build in AMD64).
Thanks so much for your help!
By mike
Date 2007-04-14 03:22
No, my binary had no mods, but now that you can build it I'll leave the mod up to you. Cheers! mike
Topic SeaDAS / SeaDAS 6.x - General Questions / missing values from MODIS chlorophyll (locked) (21343 hits)
