We have been using version 1.5 for some time and now I find the version has moved on to 1.7. So I have downloaded and manually installed and tried to integrate it into our Modis DBS processing system.
The Level0 to Level1/GEO conversion now seems to be 2 separate scripts. Running modis_L1A.csh is successful but it fails in modis_GEO.csh. Looking at the logs the problem seems to be "wrong version
of Toolkit installed". Don't really understand how this part of the system works, but it is apparently looking for Toolkit version TK5.2.17 and our version is TK.5.2.12, from our earlier installation. This seems to be a hard-coded requirement in the program 'geogen'.

Do I need to install a new version of the Toolkit somehow? Where is the Toolkit software installed anyway? Is there an environment variable I have missed to control this? Or maybe I have to recompile the program 'geogen' to use the right Toolkit? I installed  'geogen' as a binary with MODISL1DB 1.7.  Could I use the old 'geogen' program from our earlier version 1.5?

Ian Brown

The logs are saying the toolkit version is wrong. But analysing it further it fails in modis_GEO.csh when running modis_timestamp. I have tried this manually on the MOD01....hdf file produced by modis_L1A.csh,  and it is modis_timestamp which is returning the errror:
"modis_timestamp: Error - File doesn't appear to have a valid MODIS HDF header"
This seems to be what is causing modis_GEO.csh to fail. MOD01 file was created from one of our locally-received DBS Level 0 files - a .isp file named "clear.......isp".  These process OK under MODISL1DB version 1.5 which we normally use.

Thanks

One further post! The MODISL1DB 1.5 version of 'modis_timestamp' seems to give a correct time response when run against the MOD01 file created by modis_L1A.csh version 1.7. So it appears to be the new version of modis_timestamp in 1.7 which is causing my problem.

What OS are you running?

Sean

Sean, we are running on Red Hat Enterprise Linux. MODISL1DB version 1.5 has been running successfully under this OS for quite some time. Thanks

Tried running modis_timestamp in a stand-alone C Shell script and got the following error

modis_timestamp: Error - ncdump must exist in $SEADAS/src/lib3/hdf4/bin/ or $SEADAS/bin/ to run this program.

Our $SEADAS variable is set to

/data/radsat/frpd/polar/modis/modisl1db_1.7

ie the installation directory. But the directory containing ncdump is $SEADAS/run/bin not $SEADAS/bin . Is this the problem?

Ian,

Yes, that would be a problem...However, this suggests you're using an old version of modis_timestamp.
The one distributed with modisl1db_1.7/modisl1db_linux.tar.gz
no longer points to src/lib3/hdf4/bin for ncdump.

Does a cksum on your modis_timestamp yield the following?:

> cksum run/bin/modis_timestamp

613192076 1086741 run/bin/modis_timestamp

Can you confirm that you're not somehow pointing to the version distributed with 1.5?

Sean

Sean, have been trying various things here so maybe I have confused the issue. Yes that error message came from the 1.5 version of modis_timestamp; but only because I had copied it temporarily into the /bin directory for 1.7, as a test.  I have now re-installed the 1.7 version of modis_timestamp and am now getting the original error once again:

modis_timestamp: Error - File doesn't appear to have a valid MODIS HDF header.

Sorry for the confusion. Ian

Sean, to recap on the problem and hopefully remove earlier confusion (sorry):

It is failing when script modis_GEO.csh runs  modis_timestamp against the MOD01.....hdf file:   'modis_timestamp MOD01....hdf  start' and gives the error message that
this file doesn't have a valid Modis HDF Header. The MOD01....hdf file was produced by the modis_L1A.csh (1.7) script, which appears to run successfully. This has been
run with several different AQUA and TERRA passes, and seems to fail the same way in all cases.

Ian

Ian,

The problem might yet be a path issue...I've just realized that the $DBHOME/run/script/modisl1db_env.[c|ba]sh scripts
do not properly define the path to the script and bin directories.  These should be $DBHOME/run/scripts and $DBHOME/run/bin
respectively, but are $DBHOME/scripts and $DBHOME/bin.

Could you try modifying your path appropriately, and see it that gets you farther?

I'll reissue the 1.7 release with this path issue fixed, as well as the fix to a modis_GEO.csh issue that was discovered with the latest SeaDAS release - which may not affect you as your using an OS we tested against.

Sean

Sean, thanks. Not sure that the path is the problem. I did notice it was apparently wrong, so I changed it as shown below. But when I ran it again the same happened as before: modis_L1A.csh ran OK and created a MOD01...hdf file; then modis_GEO.csh ran against this file and failed in the same way:  this was our command and the error message from our log.

++++

/data/radsat/frpd/polar/modis/modisl1db_1.7/run/scripts/modis_GEO.csh /data/radsat/frpd/polar/modis/level1/MOD01_terra_58980_20110119_113840.hdf -o /data/radsat/frpd/polar/modis/level1/MOD03_terra_58980_20110119_113840.hdf   -geocheck_threshold 50 -save-log
modis_GEO.csh: ERROR: Unable to determine start time from L1A file.

++++

This is the modisl1db_env.csh as we now have it:

++++

setenv SEADAS $DBHOME
setenv OCSSWROOT $DBHOME
setenv OCDATAROOT $DBHOME/run/data
setenv MODIS_GEO .
setenv MODIS_L1A .
setenv MODIS_L1B .
setenv MODIS_ATTEPH $DBHOME/run/var/modis/atteph
setenv AQUA_REFL_LUT $DBHOME/run/var/modisa/cal/OPER/MYD02_Reflective_LUTs.V6.1.5.0g.hdf
setenv AQUA_EMIS_LUT $DBHOME/run/var/modisa/cal/OPER/MYD02_Emissive_LUTs.V6.1.5.0g.hdf
setenv AQUA_QA_LUT $DBHOME/run/var/modisa/cal/OPER/MYD02_QA_LUTs.V6.1.5.0g.hdf
setenv TERRA_REFL_LUT $DBHOME/run/var/modist/cal/EVAL/MOD02_Reflective_LUTs.V6.1.2.1.hdf
setenv TERRA_EMIS_LUT $DBHOME/run/var/modist/cal/EVAL/MOD02_Emissive_LUTs.V6.1.2.1.hdf
setenv TERRA_QA_LUT $DBHOME/run/var/modist/cal/EVAL/MOD02_QA_LUTs.V6.1.2.1.hdf
set path=(. $DBHOME/run/scripts $DBHOME/run/bin $path)

++++

We are setting $DBHOME to: /data/radsat/frpd/polar/modis/modisl1db_1.7  - ie the 'root' installation directory.

++++

From what you say I did not need to apply your update_01 patch. However I downloaded and tried it anyway, and got the following error on that:

++++

Checking MODISL1DB Set Up...

Installing MODISL1DB 1.7 Update 01...
OCSSW_ARCH: Undefined variable.

++++

We are still using version 1.5 operationally, so this is only to evaluate 1.7. I would like to install 1.7 permanently as it may fix some problems we have, or may run faster.

Many thanks, Ian

I tried running modis_timestamp again as short csh script:

++++

#!/bin/csh -f

setenv DBHOME /data/radsat/frpd/polar/modis/modisl1db_1.7

source ${DBHOME}/run/scripts/modisl1db_env.csh

echo $SEADAS

/data/radsat/frpd/polar/modis/modisl1db_1.7/run/bin/modis_timestamp MOD01_terra_58980_20110119_113840.hdf start

++++

This gives the same results as before:
++++

/data/radsat/frpd/polar/modis/modisl1db_1.7       ($SEADAS setting)
modis_timestamp: Error - File doesn't appear to have a valid MODIS HDF header.

++++

So it  is failing as above when modis_GEO.csh runs the modis_timestamp program. I think our shell variables must be OK now as it seems to be finding everything in the right place, including the
$SEADAS/run/bin and $SEADAS/run/scripts directories.

Maybe it is modis_timestamp which is looking in the wrong place, or is there really something wrong with our MODO1...hdf file? (produced by modis_L1A.csh). Maybe it doesn't like our filename?

Hope this throws more light on the problem. Thanks

Ian

Ian,

  1) I did bugger the update - a revised one is now available that corrects the OCSSW_ARCH problem you found.

  2) Could you run a cksum on the modis_timestamp binary?  should get the following:

      cksum $DBHOME/run/bin/modis_timestamp 
      613192076 1086741 /modisl1db/run/bin/modis_timestamp
 
  3) Could you run ncdump -h on the L1A file? Does the output look correct (see attached example)?

  4) Could you grab the seadas_benchmarks.tar.gz, explode the tarfile and run the seadas_benchmarks.bash script?
   This will process a L0 file through L1B (for a modisl1db install, does more if you have the full SeaDAS)
   - if you don't have the dem files (seadas_dem_modis.tar.gz) you'll either need them or need to modify the seadas_benchmarks.bash script to
     add '-disable-dem' to the modis_GEO.csh call

    IF the benchmark script completes with no errors, then your install should be OK and it might be your file (or something else more insidious...)

Sean

Sean,  I initially tried the cksum on modis_timestamp and it came out as expected:

radsat-3,polar: cksum modis_timestamp
613192076 1086741 modis_timestamp

Sean, have now tried ncdump on our MOD01 file and got a floating point exception:

Did it in the following script to set up $DBHOME etc

++++

#!/bin/csh -f

setenv DBHOME /data/radsat/frpd/polar/modis/modisl1db_1.7

source ${DBHOME}/run/scripts/modisl1db_env.csh

echo $SEADAS

/data/radsat/frpd/polar/modis/modisl1db_1.7/run/bin/ncdump MOD01_terra_58980_20110119_113840.hdf

++++

and gave the following output

++++

radsat-3,polar: ncd
/data/radsat/frpd/polar/modis/modisl1db_1.7
Floating exception

++++

I wonder if ncdump is relying on any HDF shared library which it can't find?  Or is the HDF library compiled in with the binary? Ian

Sean,

Tried running the 'lonlat2pixline' program with our MOD01 file. This seems to have trouble reading the file also, and gave errors as below:

++++

-E- getformat.c Line 347: Unrecognized HDF-EOS file MOD01_terra_58980_20110119_113840.hdf
-E- lonlat2pixline: File type not recognized for MOD01_terra_58980_20110119_113840.hdf.
radsat-3,polar:                     

++++

So looks like the problem may be with the file itself rather than with ncdump. It was created using the script modis_L1A.csh, and this appears to run successfully with no error messages.

Or is this another red herring!

Ian

Ian,

ncdump and lonlat2pixline will both use the HDF libaries, so it's either
the file or HDF... we recently discovered an issue with HDF where the
libz it's built against is a shared object, rather than the static library.
If your system doesn't have libz (hard to imagine as it seems rather
prolific) you may have troubles with any program built against HDF.

Could you try running: 
    ldd $DBHOME/run/bin/ncdump

or even step back and do a 'file' on your L1A file:
    file MOD01_terra_58980_20110119_113840.hdf

Sean

Sean, I think we might be getting to the root of this problem now.  The ldd command on ncdump version 1.7 gives an error, while ldd on ncdump version 1.5 shows a dependence on shared libraries:

++++

radsat-3,polar: ldd ncdump.version1.7
/usr/bin/ldd: line 124: 19104 Floating point exceptionLD_TRACE_LOADED_OBJECTS=1 LD_WARN= LD_BIND_NOW= LD_LIBRARY_VERSION=$verify_out LD_VERBOSE= "$@"

radsat-3,polar: ldd ncdump.version1.5
        libm.so.6 => /lib/tls/libm.so.6 (0x0084b000)
        libc.so.6 => /lib/tls/libc.so.6 (0x0071f000)
        /lib/ld-linux.so.2 (0x00706000)

++++

The 1.5 version of ncdump gives sensible output from our MOD01 file, while the 1.7 version fails with a 'Floating Point Exception' as before/above.

So I tried copying ncdump version 1.5 into the MODISL1B version 1.7 ..run/bin directory. The modis_GEO.csh script now progresses beyond modis_timestamp and seems to create a MOD03 GEO file.
However it then fails when checking the MOD03 file using the script modis_geocheck.csh.  This gives the following output:

++++

*Use of definitive attitude and ephemeris disabled*
Input  Level 1A   : ./MOD01_terra_58980_20110119_113840.hdf
Output Geolocation: ./MOD03.hdf

Satellite: terra
Year: 2011  Day: 019  Hour: 11  Minute: 40
*Terrain Correction Enabled*

Creating MODIS geolocation file...
/data/radsat/frpd/polar/modis/modisl1db_1.7/run/bin/geogen_modis
GEO version: 5.0.14 built on Dec 23 2010 (12:16:59)
geogen_modis exit status: 1

Running validation test on geolocation file...
/data/radsat/frpd/polar/modis/modisl1db_1.7/run/scripts/modis_geocheck.csh MOD03.hdf 50
*** ERROR: geogen_modis failed to produce a geolocation file.
*** Validation test failed for geolocation file MOD03.hdf.

modis_GEO.csh: ERROR: MODIS geolocation processing failed.
Please ensure utcpole.dat and leapsec.dat are up-to-date in
/data/radsat/frpd/polar/modis/modisl1db_1.7/run/var/modis
Please examine the LogStatus and LogUser files for more information.

++++

I see that modis_geocheck.csh also relies on 'ncdump' to get the dimensions of the file, so maybe there is a similar problem here - will investigate that further.

But the main issue seems to be that the 1.7 version of ncdump fails on our system, while the 1.5 version works OK, although it seems to be dependent on some shared libraries we have installed here.

What do you suggest I do about this? Think we are making some progress now. 

Many thanks, Ian

This is the log from modis_GEO.sh which I ran with some new TERRA data from this morning. The geolocation check may have failed before because I used old data.

This was using the 1.5 version of ncdump instead of the 1.7 version supplied. Everything else was version 1.7 as installed.

You will see it creates the MOD01 file OK using modis_L1A.sh, but then once again fails in modis_GEO.sh. But the error messages still relate to ncdump, apparently coming from the modis_geogen program.

So ncdump seems to be the main cause of our problems here. Hope that helps.

Thanks again, Ian

This is running modis_GEO.csh against our MOD01 file.  Contrary to earlier post it appears NOT to be creating the MOD03 file at all, because geogen_modis is  giving exit status 1.
ncdump problem inside geogen_modis?

++++

radsat-3,polar: rgeo
*Use of definitive attitude and ephemeris disabled*
Input  Level 1A   : ./MOD01_terra_58994_20110120_104339.hdf
Output Geolocation: ./MOD03.hdf

Satellite: terra
Year: 2011  Day: 020  Hour: 10  Minute: 45
*Terrain Correction Enabled*

Creating MODIS geolocation file...
/data/radsat/frpd/polar/modis/modisl1db_1.7/run/bin/geogen_modis
GEO version: 5.0.14 built on Dec 23 2010 (12:16:59)
geogen_modis exit status: 1

Running validation test on geolocation file...
/data/radsat/frpd/polar/modis/modisl1db_1.7/run/scripts/modis_geocheck.csh MOD03.hdf 50
*** ERROR: geogen_modis failed to produce a geolocation file.
*** Validation test failed for geolocation file MOD03.hdf.

modis_GEO.csh: ERROR: MODIS geolocation processing failed.
Please ensure utcpole.dat and leapsec.dat are up-to-date in
/data/radsat/frpd/polar/modis/modisl1db_1.7/run/var/modis
Please examine the LogStatus and LogUser files for more information

++++

Sean, sorry for all the posts but I am trying things on the fly here.

I finally made modis_GEO.sh work! It successfully created a MOD03 file using the MOD01 file created earlier by modis_L1A. It was failing before because I forgot to disable terrain correction.

See latest attachment for log.

This was using the MODISL1DB version 1.7 installation, except that I have inserted the 1.5 version of ncdump in place of the installed 1.7 version.

So the real problem is still: why does the installed 1.7 version of ncdump not work for us?

Ian

Ian,

This seems to stem from an update to libz that we implemented when we updated all the third party libraries
necessary for our processing code.  Apparently, we let a shared object slip in...so, while for MOST cases this
was innocuous - in that when the binary was run it searched for the .so in the usual places, it usually found one
in /usr/lib.  Thus we never noticed there was an issue.  We are currently testing a revised build that should
fix this issue.  Well, that's the hope anyway

In the meantime, you should be OK with using the ncdump from the earlier release.
But, please let us know if you find any other issues downstream.

Sean

Sean. I am glad we got to the bottom of it at last.

Meanwhile I am quite happy to continue testing with the 1.5 version of ncdump, which you say should be OK.

Apart from this issue it looks like the MODISL1DB version 1.7 installation is starting to work for us now. So I aim to replace version 1.5 with 1.7 as soon as I am confident there are no more problems.

Many thanks for all the help and guidance.

Regards, Ian

When I run modis_GEO.csh against the MOD01 file in isolation it works fine.

So I tried running modis_L1a.csh and modis_GEO.sh automatically using our regular level 1 processing scripts and got another ncdump-related error:

++++

Scan Number: 240  Thu Jan 20 16:18:38 2011
Scan Number: 250  Thu Jan 20 16:18:38 2011
Scan Number: 260  Thu Jan 20 16:18:38 2011
Scan Number: 270  Thu Jan 20 16:18:38 2011
Scan Number: 280  Thu Jan 20 16:18:39 2011
Scan Number: 290  Thu Jan 20 16:18:39 2011
Scan Number: 300  Thu Jan 20 16:18:39 2011
Scan Number: 310  Thu Jan 20 16:18:40 2011
Scan Number: 320  Thu Jan 20 16:18:40 2011
Scan Number: 330  Thu Jan 20 16:18:40 2011
Scan Number: 340  Thu Jan 20 16:18:40 2011
Scan Number: 350  Thu Jan 20 16:18:41 2011
Scan Number: 360  Thu Jan 20 16:18:41 2011
Scan Number: 370  Thu Jan 20 16:18:42 2011
Scan Number: 380  Thu Jan 20 16:18:42 2011
Scan Number: 390  Thu Jan 20 16:18:42 2011
Scan Number: 400  Thu Jan 20 16:18:43 2011
Scan Number: 410  Thu Jan 20 16:18:43 2011
Scan Number: 420  Thu Jan 20 16:18:43 2011
Scan Number: 430  Thu Jan 20 16:18:43 2011
Scan Number: 440  Thu Jan 20 16:18:44 2011
l1agen_modis exit status: 0
MODIS L1A processing complete.

/data/radsat/frpd/polar/modis/modisl1db_1.7/run/scripts/modis_GEO.csh /data/radsat/frpd/polar/modis/level1/MOD01_terra_58994_20110120_104339.hdf -o /data/radsat/frpd/polar/modis/level1/MOD03_terra_58994_20110120_104339.hdf   -disable-definitive -disable-dem -geocheck_threshold 50 -save-log
*** /data/radsat/frpd/polar/modis/modisl1db_1.7/run/bin/ncdump: ncopen failed on /data/radsat/frpd/polar/modis/level1/MOD01_terra_58994_20110120_104339.hdfåi
modis_GEO.csh: ERROR: Unable to determine start time from L1A file.

16:18:45 Level 0 file: clear_TER_58994_20-JAN-2011_10:43:39.461.isp Level 1a processing failure

++++

If I do ncdump -h on the MOD01 file it also works OK. Not sure why this happens but it is obviously the same problem.

Just some more debug information for you. Could you let me know please when you have a solution to the ncdump problem.

Thanks again, Ian

Just noticed that it seems to introduce spurious characters at end of the filename, when ncdump tries to do the ncopen, so that's probably why it fails?

Sean,

Just to let you know there are no further problems here with MODISL1DB version 1.7.

I have this morning successfully run all stages of processing (L1A, GEO and L1B) against one of our locally-received TERRA passes.

This is version 1.7 as supplied in your tar file (Linux version), except that I replaced the 1.7 version of ncdump with the older version from 1.5.
The 1.7 version was consistently failing, while the 1.5 version seems to work fine with the rest of the 1.7 software.

There may be a minor issue with ncdump (1.5) failing on long file names during 'ncopen' (see posts above), but I was able to get round that OK.

So quite happy with MODISL1DB version 1.7 now. Thanks for all the help. Regards, Ian

Ian,

Glad it's working for you

If you have a free moment, could you grab a rebuilt ncdump and give it a try?

Before I commit to another update, I want to be sure that the issue you discovered
is resolved.

The file will be available at: ftp://samoa.gsfc.nasa.gov/pub/seadas/ncdump
until you have a chance to respond.

Thanks,
Sean

Hi Sean, I downloaded your latest version of ncdump from the above ftp link. But it's apparently producing the same error as before on our system:

++++

radsat-3,polar: ncdump.1.7.updated
Floating point exception
radsat-3,polar: ldd ncdump.1.7.updated
/usr/bin/ldd: line 124: 13759 Floating point exceptionLD_TRACE_LOADED_OBJECTS=1 LD_WARN= LD_BIND_NOW= LD_LIBRARY_VERSION=$verify_out LD_VERBOSE= "$@"
radsat-3,polar:       

++++                                                                                                                               

This happens immediately, whatever arguments I supply. The ldd command on the file is still producing the same error as before also.

So still using the 1.5 version of ncdump successfully here.

Ian

This is the environment where I tried it: your new version was copied into file ncdump.1.7.updated in order to test it.

++++
radsat-3,polar: pwd
/data/radsat/frpd/polar/modis/modisl1db_1.7/run/bin
radsat-3,polar: ll
total 44724
-rwxrwxr-x  1 polar msg  551474 Dec 24 15:48 addsecs
-rwxrwxr-x  1 polar msg  549633 Dec 24 15:48 date2jd
-rwxrwxr-x  1 polar msg 7195302 Dec 24 15:48 geogen_modis
-rwxrwxr-x  1 polar msg  549601 Dec 24 15:48 jd2date
-rwxrwxr-x  1 polar msg  651306 Dec 24 15:48 l0chunk_modis
-rwxrwxr-x  1 polar msg  684951 Dec 24 15:48 l0cnst_write_modis
-rwxrwxr-x  1 polar msg  646187 Dec 24 15:48 l0fix_modis
-rwxrwxr-x  1 polar msg 1946658 Dec 24 15:48 l1aextract_modis
-rwxrwxr-x  1 polar msg 6138213 Dec 24 15:48 l1agen_modis
-rwxrwxr-x  1 polar msg 1960134 Dec 24 15:48 l1asubset_modis
-rwxrwxr-x  1 polar msg 6574489 Dec 24 15:48 l1bgen_modisa
-rwxrwxr-x  1 polar msg 6569465 Dec 24 15:48 l1bgen_modist
-rwxrwxr-x  1 polar msg 5465041 Dec 24 15:48 lonlat2pixline
-rwxrwxr-x  1 polar msg 1086741 Jan 20 14:41 modis_timestamp
-rwxr-xr-x  1 polar msg  491456 Jan 20 14:33 modis_timestamp.version1.5
-rwxr-xr-x  1 polar msg 1086741 Jan 13 17:09 modis_timestamp.version1.7
-rwxr-xr-x  1 polar msg  478084 Jan 20 10:44 ncdump
-rwxr-xr-x  1 polar msg 1293677 Jan 26 15:53 ncdump.1.7.updated
-rwxr-xr-x  1 polar msg  478084 Jan 20 10:42 ncdump.version1.5
-rwxr-xr-x  1 polar msg 1236937 Jan 20 10:43 ncdump.version1.7
radsat-3,polar: 

radsat-3,polar: ncdump.1.7.updated
Floating point exception
radsat-3,polar: ldd ncdump.1.7.updated
/usr/bin/ldd: line 124: 13759 Floating point exceptionLD_TRACE_LOADED_OBJECTS=1 LD_WARN= LD_BIND_NOW= LD_LIBRARY_VERSION=$verify_out LD_VERBOSE= "$@"
radsat-3,polar: ncdump.1.7.updated -h
Floating point exception

+++

                                                              

Ian,

Thanks.  I was hoping the shared object problem was with libz (the problem we fixed) - apparently it is
lower level than that.  I suspect that your version of glibc is simply not compatible with the version we
have on our CentOS5.5 system (the one I built against for this release).

Sean