I've just started working with MODIS Terra data. I've got several questions related to data access and usage.

1. I'd like to automate the process of downloading refined data. I want to make sure that all the data for a particular day is available before I process it. Is there either a way to tell when all the data for a day is available or a number of days after which all the data for a day is guaranteed to be available?

2. How long is data kept in the subscriptions area (the subscription I will be downloading from is 89)? I'd like to avoid downloading files more than once and I definitely don't want to process files more than once. I'm hoping understanding how files appear and disappear from the directory will help me automate the processing.

3. One of the existing applications I'll be modifying accumulates data for several days and averages it. Currently the application uses the data directly from the files. The slope/intercept data is not used. Is this safe? I know there are potential overflow issues but I'm also concerned that the slope/intercept (as well as other metadata) may not be identical in all files. Is it safe to assume that the metadata is identical in all files?

Thanks in advance.

Ken Fairchild

Ken,

First let me say that  we are not yet processing MODIS Terra data.  We DO have the entire mission for MODIS Aqua, though, and these are available via subscription.  There are 288 granules per day, with three states: Day, Night, and Mixed.  If you are processing ALL data for a day, you should expect 288 files.  However, if you are only interested in the daytiime granules, the number will vary about an average of 130.  These numbers are for the entire globle.  Assuming your subscription is not for the entire globle, and if your subscription is set for refined processing,  you can expect that all available granules for a given day have been received and processed.   There will be exceptions to this, but in general by the time the refined processing is kicked off, we have all the data we'll get.

Data are kept in the subscription area for 3 days.  To avoid downloading files multiple times, try using an ftp client like ncftp, which by default will not download files already retrieved.

Product specific metadata will not vary between files, so the slope and intercept for a product in one file will be the same in all files.

Regards,
Sean

Ken,

Regarding the wait period for refined data, the refined processing currently uses ancillary data from 4 different sources:

definitive attitude & ephemeris data from the DAAC
meteorological data from the NCEP project
ozonic data from the EPTOMS project
SST data from NOAA

Each MODIS granule matches up with an optimal set of the above data.  Of them, the SST data can take the longest to arrive, up to 9 days after a granule's observation time.  Our system waits up to 10 days for the optimal ancillary data to arrive.  After that, the best-available set is selected.

So under normal operations, all of the refined data will have been staged to the subscription directory within 11 days.

Regards,
john

With regard to the slope and intercept meta-data, I would caution that it can change.  Any change would normally be tied to a reprocessing, but in such cases the data in the forward stream would be changed as well.  We may in fact be changing the nLw_667 scaling to avoid saturating the 16-bit integer in extremely turbid water.  I would advise that you read and apply these scaling factors as a matter of routine.

-- bryan