Hi.  I'm doing some MODIS processing using SeaDAS6.3 beginning with extracted L1A files.  Its been a while since I've done this, but it seems to me that the process of creating the GEO files now is taking a lot longer than it used to (certainly could be my imagination or lack of patience).   The set of extracted files I'm working with are relatively small (~50 scans each) and its taking a little over two minutes (clock time) per file to create the GEO file.  In contrast, going from the L1A/GEO files to L1B takes about 15 seconds per file.  I know its going to take whatever time it takes, but I was just curious to know if there might be some reason that new version would be slower than the old.   Thanks, Barry

Barry,

I'm always hesitant to give the 'works for me' response, as if one person sees an issue, it may be more wide spread.
However, in this case, after an admittedly quick test on a couple of systems, I'm leaning toward 'It's just you'

I was able to run modis_GEO.py on an extract of 120 scans lines in 5.6 seconds on a Mac laptop and 3.3 seconds on
a 64bit Ubuntu desktop.

If you run one a second time, does it take the same 2 minutes?

Sean

The processing executables haven't changed.  Do the logs (or screen output) give you any indication of what's taking the most time?

The first run on any system will take a little while - it needs to build a system-specific version of the planetary ephemeris file $SEADAS/run/data/modis/static/de200.eos.

Querying our database to obtain appropriate spacecraft attitude and ephemeris files could also be a bottleneck, if you don't already have them locally.

Thanks guys.  Yes, I'm inclined to agree that its 'just me.'  Turns out I've got other problems with MODIS_geo that I'm trying to resolve - strings of files with 0% valid pixels - strange intermittent python error messages, etc.  I'm doing a reinstall this morning and will try again.  I'll let you know if that fixes things.

Hi - It looks like my problem with slow execution of MODIS_geo might be due to a combination of factors - I happened to be doing a long data download using another process this morning and that seemed to slow things.  In addition I seem to have run into a string of L1A files that result in "Percent valid data (0.00) is less than threshold (95.00)."  Processing these files takes much longer than ones that seem OK.  Now, the question is why am I running into so many successive files that can't pass the validation test.  I'm doing my processing in batch, of course, but I went back to the GUI to check the original file and to try generating the GEO file interactively.  The original L1A file looks to be OK (at least I can display it).  Here is the output from the attempt to create the GEO file.

SeaDAS Version 6.3 (pid = 4620)
SeaDAS>
unix cmd =
$SEADAS/run/scripts/modis_GEO.py /Volumes/2TBDrive/MODIS/L1/A2003111181000.L1A_LAC.x.hdf -o /Volumes/2TBDrive/MODIS/L1/A2003111181000. EO --verbose  --threshold=95
Determining required attitude and ephemeris files...
Searching database: /Volumes/1TBSATDATA/sw/seadas6.3/log/ancillary_data.db

Input file: /Volumes/2TBDrive/MODIS/L1/A2003111181000.L1A_LAC.x.hdf
Start time: 2003111181000
End time: 2003111181459

  Found: PM1EPHND.P2003111.1200.001
  Found: PM1ATTNR.P2003111.1800.003

Creating MODIS geolocation file...
GEO version: 5.0.14 built on Oct 31 2011 (13:08:01)
scan: 0 out of 47 Wed Dec  7 09:45:28 2011
scan: 10 out of 47 Wed Dec  7 09:45:31 2011
scan: 20 out of 47 Wed Dec  7 09:45:33 2011
scan: 30 out of 47 Wed Dec  7 09:45:36 2011
scan: 40 out of 47 Wed Dec  7 09:45:38 2011
geogen_modis returned with exit status: 1
Will run modis_geocheck to confirm results...

Running validation test on geolocation file..
Percent valid data (0.00) is less than threshold (95.00)

exit_status= 

My script still is running and so far these files have had the 0% valid pixels.  Any idea why?

LogUser.A2003111181000.GEO.x.hdf
LogUser.A2003112185000.GEO.x.hdf
LogUser.A2003113175500.GEO.x.hdf
LogUser.A2003114184000.GEO.x.hdf
LogUser.A2003115174500.GEO.x.hdf
LogUser.A2003118181500.GEO.x.hdf
LogUser.A2003119190000.GEO.x.hdf
LogUser.A2003120180500.GEO.x.hdf
LogUser.A2003121184500.GEO.x.hdf
LogUser.A2003122175000.GEO.x.hdf
LogUser.A2003123183500.GEO.x.hdf
LogUser.A2003124174000.GEO.x.hdf
LogUser.A2003125182000.GEO.x.hdf
LogUser.A2003126190500.GEO.x.hdf
LogUser.A2003127181000.GEO.x.hdf
LogUser.A2003128185000.GEO.x.hdf
LogUser.A2003129175500.GEO.x.hdf
LogUser.A2003130184000.GEO.x.hd

I had no trouble producing the (unextracted) A2003120180500.GEO.

Did you update the leapsec.dat and utcpole.dat files?  There is no longer a separate button for that in the GUI; use Update->Update Aqua calibration LUTs.  Or enter on the command line: update_luts.py aqua --verbose

If that's not the problem, please attach a sample set of Log* files for one granule.

Hi Gwyn,

Yes, I did the update.  I'll post then attach a sample of the log files.  Strange that it seems to happen often but not always.  Barry

The LogStatus file shows the following error message 86 times (prob. one per scan):

PGS_EPH_EphemAttit():PGSEPH_E_NO_SC_EPHEM_FILE:44547
TOOLKIT error
GEO_get_T_inst2ecr.c, GEO_get_T_inst2ecr():MODIS_E_GEO:288779785
Wed Dec  7 10:53:34 2011
Error returned by function PGS_CSC_quatRotate(sun vector)
GEO_interp_ECR.c, GEO_interp_ECR():MODIS_E_GEO:288779785
Wed Dec  7 10:53:34 2011
Error returned by function GEO_get_T_inst2ecr(2003-10-10T18:35:27.519068Z)
GEO_locate_one_scan.c, GEO_locate_one_scan():MODIS_E_GEO:288779785
Wed Dec  7 10:53:34 2011
Error returned by function GEO_interp_ECR(0)
PGS_EPH_EphemAttit():PGSEPH_E_NO_SC_EPHEM_FILE:44547

The "sun vector" depends on the binary planetary ephemeris - I wonder if there could be a mismatch with your system type.  Please post the results of the following:
uname -a ; ls -l $SEADAS/run/data/modis/static/de200.eos `which geogen_modis`

First, thanks again, Gwyn for helping me with this.  Here is the result of the command you suggested.  Looks

bash-3.2$ uname -a ; ls -l $SEADAS/run/data/modis/static/de200.eos `which geogen_modis`
Darwin beaufort.local 10.8.0 Darwin Kernel Version 10.8.0: Tue Jun  7 16:33:36 PDT 2011; root:xnu-1504.15.3~1/RELEASE_I386 i386
-rwxr-xr-x  1 blesht  blesht  5859428 Oct 31 12:08 /Volumes/1TBSATDATA/sw/seadas6.3/run/bin/macosx_intel/geogen_modis
-rw-r--r--  1 blesht  blesht  4764304 Dec  6 10:55 /Volumes/1TBSATDATA/sw/seadas6.3/run/data/modis/static/de200.eos

Also, I don't know if it helps any, but this problem began with A20031101905 (all of 2002 and 2003 up to day 110 were fine).  For the remainder of 2003 (~250 files), it has occurred about 100 times.

Barry

Barry -

It looks like everything is in sync.  Since you reinstalled after the de200.eos file was originally built, you may want to delete it and let it rebuild.  You might also check that your attitude and ephemeris files are about the same size as those you processed with successfully.

Other that that, I'm out of ideas.
- Gwyn

Well - you might have hit on it.  Look at the following-

The first set are the ephemeris files for those granules that failed - except for days 116 and 117 which apparently were OK (no log files generated), despite the 0-size PM1EPHND files.

The second set are the ephemeris files for the previous 10 days which all seemed fine.  Note that each days files are the same size.  Now the question, of course, is why is that? 

Also, I'm not sure what you mean by "letting it rebuild" with respect to the de200.eos file, but maybe we should sort out the ephemeris file problem first.

Barry

bash-3.2$ ls -l 11*/P*.*
-rw-r--r--  1 blesht  blesht   278528 Dec  6 22:02 110/PM1ATTNR.P2003110.1800.003
-rw-r--r--  1 blesht  blesht  3997696 Dec  6 16:10 110/PM1EPHND.P2003110.1200.001
-rw-r--r--  1 blesht  blesht   294912 Dec  6 22:04 111/PM1ATTNR.P2003111.1800.003
-rw-r--r--  1 blesht  blesht    49152 Dec  6 16:11 111/PM1EPHND.P2003111.1200.001
-rw-r--r--  1 blesht  blesht   147456 Dec  6 22:04 112/PM1ATTNR.P2003112.1800.003
-rw-r--r--  1 blesht  blesht   393216 Dec  6 16:12 112/PM1EPHND.P2003112.1200.001
-rw-r--r--  1 blesht  blesht    32768 Dec  6 22:05 113/PM1ATTNR.P2003113.1600.003
-rw-r--r--  1 blesht  blesht    32768 Dec  6 16:12 113/PM1EPHND.P2003113.1200.001
-rw-r--r--  1 blesht  blesht   463104 Dec  6 22:06 114/PM1ATTNR.P2003114.1800.003
-rw-r--r--  1 blesht  blesht   770048 Dec  6 16:15 114/PM1EPHND.P2003114.1200.001
-rw-r--r--  1 blesht  blesht   463104 Dec  6 22:09 115/PM1ATTNR.P2003115.1600.003
-rw-r--r--  1 blesht  blesht   163840 Dec  6 16:15 115/PM1EPHND.P2003115.1200.001
-rw-r--r--  1 blesht  blesht   463104 Dec  6 22:11 116/PM1ATTNR.P2003116.1800.003
-rw-r--r--  1 blesht  blesht        0 Dec  6 16:16 116/PM1EPHND.P2003116.1200.001
-rw-r--r--  1 blesht  blesht    65536 Dec  6 22:11 117/PM1ATTNR.P2003117.1800.003
-rw-r--r--  1 blesht  blesht        0 Dec  6 16:16 117/PM1EPHND.P2003117.1200.001
-rw-r--r--  1 blesht  blesht   463104 Dec  6 22:13 118/PM1ATTNR.P2003118.1800.003
-rw-r--r--  1 blesht  blesht    65536 Dec  6 16:17 118/PM1EPHND.P2003118.1200.001
-rw-r--r--  1 blesht  blesht   212992 Dec  6 22:14 119/PM1ATTNR.P2003119.1800.003
-rw-r--r--  1 blesht  blesht  1785856 Dec  6 16:22 119/PM1EPHND.P2003119.1200.001
bash-3.2$ ls -l 09*/P*.*
-rw-r--r--  1 blesht  blesht   463104 Dec  6 15:47 090/PM1ATTNR.P2003090.1600.003
-rw-r--r--  1 blesht  blesht  5531136 Dec  6 15:47 090/PM1EPHND.P2003090.1200.001
-rw-r--r--  1 blesht  blesht   463104 Dec  6 15:48 091/PM1ATTNR.P2003091.1800.003
-rw-r--r--  1 blesht  blesht  5531104 Dec  6 15:48 091/PM1EPHND.P2003091.1200.001
-rw-r--r--  1 blesht  blesht   463104 Dec  6 15:49 092/PM1ATTNR.P2003092.1600.003
-rw-r--r--  1 blesht  blesht  5531136 Dec  6 15:49 092/PM1EPHND.P2003092.1200.001
-rw-r--r--  1 blesht  blesht   463104 Dec  6 15:50 093/PM1ATTNR.P2003093.1800.003
-rw-r--r--  1 blesht  blesht  5531104 Dec  6 15:50 093/PM1EPHND.P2003093.1200.001
-rw-r--r--  1 blesht  blesht   463104 Dec  6 15:51 094/PM1ATTNR.P2003094.1800.003
-rw-r--r--  1 blesht  blesht  5531136 Dec  6 15:51 094/PM1EPHND.P2003094.1200.001
-rw-r--r--  1 blesht  blesht   463104 Dec  6 15:53 095/PM1ATTNR.P2003095.1800.003
-rw-r--r--  1 blesht  blesht  5531104 Dec  6 15:53 095/PM1EPHND.P2003095.1200.001
-rw-r--r--  1 blesht  blesht   463104 Dec  6 15:54 096/PM1ATTNR.P2003096.1800.003
-rw-r--r--  1 blesht  blesht  5531136 Dec  6 15:54 096/PM1EPHND.P2003096.1200.001
-rw-r--r--  1 blesht  blesht   463104 Dec  6 15:55 097/PM1ATTNR.P2003097.1600.003
-rw-r--r--  1 blesht  blesht  5531136 Dec  6 15:55 097/PM1EPHND.P2003097.1200.001
-rw-r--r--  1 blesht  blesht   463104 Dec  6 15:56 098/PM1ATTNR.P2003098.1800.003
-rw-r--r--  1 blesht  blesht  5531104 Dec  6 15:56 098/PM1EPHND.P2003098.1200.001
-rw-r--r--  1 blesht  blesht   463104 Dec  6 15:57 099/PM1ATTNR.P2003099.1600.003
-rw-r--r--  1 blesht  blesht  5531136 Dec  6 15:57 099/PM1EPHND.P2003099.1200.001

Here's mine, with sizes more like your second group:
-rw-r--r-- 1 gfireman gfireman  463104 2011-12-07 11:46 PM1ATTNR.P2003120.1800.003
-rw-r--r-- 1 gfireman gfireman 5531104 2011-12-07 11:46 PM1EPHND.P2003120.1200.001

Looks like a network issue.  We don't have anything in place yet to verify download integrity.
Did you get valid GEO file outputs for day 116 & 117?  I wouldn't trust them!

As for rebuilding the de200.eos file, see:
http://oceancolor.gsfc.nasa.gov/forum/oceancolor/topic_show.pl?pid=18222#pid18222

Hi.  I haven't checked the day 116/117 GEO files yet, but I agree that they likely aren't to be trusted.  Am I correct that once an ephemeris file exists on the local system the script will not try to obtain a new one?  Maybe I should try deleting the bad ones and re-running MODIS_geo.  This may also account for some of the initial slowness that prompted my first message - if there were network problems and if the process was having trouble retrieving the files from the server it would take a while and might also result in corrupt files.  Nothing to lose, I guess by giving it a shot.  I'm confident that you've hit on the answer and again, offer my thanks.   Barry

Oh - the link about rebuilding just points back to this same thread.  Did you mean to point to a different one.  B.

Yes, best to delete those short files and try again.

I pointed to the message that states:

The first run on any system will take a little while - it needs to build a system-specific version of the planetary ephemeris file $SEADAS/run/data/modis/static/de200.eos.