getanc times out

Ocean Color Forum - Not logged in

Forum Ocean Color Home Help Search Login

Topic SeaDAS / SeaDAS 6.x - General Questions / getanc times out (locked) (2914 hits)

By kjwh Date 2012-01-12 14:20

Hello,
I have a similar problem as described http://oceancolor.gsfc.nasa.gov/forum/oceancolor/topic_show.pl?tid=4337. I have been trying to process MODIS L1A files, but the connection times out at the getanc step. I do have a slow network connection, but if I manually w get the file I have no problem downloading it. Is there any way to adjust the length of time it tries to connect before it times out? If not, could you please explain again how to use the getanc.py and modis_atteph.py scripts to determine the files that need to be downloaded. I have a very large list of L1A files and I would like to create an automated way to determine all of the ancillary data and then download them into the correct file structure. Lastly, what is the best way to determine if climatological MET and OZ files were used to process a given file? I would like to create an automated script to determine this and then reprocess if NRT data were not used.
Thank you for your assistance,
Kim

By WhiteG Date 2012-01-12 18:40

The thread you mention now has lots of information on working without the internet, but for the record, SeaDAS currently uses rather simple download code (based on urllib2) compared, e.g., to wget or curl:

http://stackoverflow.com/questions/2777116/difference-between-python-urllib-urlretrieve-and-wget

While it would be nice to see more robust work downloads in SeaDAS, there are also the cases where it would be better to disable all network access. Perhaps SeaDAS could have a mode where network access is disabled but a log file that can be processed later to download missing/outdated files is created.

By gwyn Date 2012-01-12 19:32

1. Sean might have a better answer, but you can try editing $SEADAS/run/scripts/modules/ProcUtils.py. The following line appears in two places; change the 10 to the desired number of seconds:

socket.setdefaulttimeout(10)

2. The very easiest way to automatically download the ancillary files would be to let the python scripts do it; files will be placed in $L2GEN_ANC. You can use your range start and end time in the following commands, not just for a granule:

getanc.py --start=${start_time} --stop=${end_time} modis_atteph.py --mission=aqua --start=${start_time} --stop=${end_time} modis_atteph.py --mission=terra --start=${start_time} --stop=${end_time}

If your network connection isn't stable enough for that, you'll have to run the above with the -d or --disable-download option, then use the resulting files to determine the filenames. For example:

for f in `cut -d\= -f2 ${start_time}.atteph`; do mkdir -p `dirname $f` cd `dirname $f` wget -c -nc http://oceandata.sci.gsfc.nasa.gov/cgi/getfile/`basename $f` cd - done

3. You'll need to look in the metadata. Something like this:

touch reprocess.txt for f in *.L2_LAC; do if `ncdump_hdf $f | egrep -q "(met|ozone)_climatology"` then echo $f >> reprocess.txt fi done

By kjwh Date 2012-01-13 19:34

Thank you,
Extending the timeout time in ProcUtils.py appears to have helped. Are there any other programs that have a similar timeout set that I should adjust? What about the downloads for the METOZ files and other ancillary files?
Thanks again,
Kim

By @sean Date 2012-01-13 23:55

Kim,

That timeout setting is for all http connections made by the python scripts. We'll look into making this a user-definable option.
Additionally, recent changes to the ProcUtils.py code have been put in place that will do automatic retries as well as other
modifications to make the downloads more robust. These will get out to the public in an upcoming update. No date set for
that yet, though.

Sean

By kjwh Date 2012-01-17 16:26

Thanks Sean, I will keep an eye out for the update.
Kim

Topic SeaDAS / SeaDAS 6.x - General Questions / getanc times out (locked) (2914 hits)