Ocean Color Forum - Not logged in
Hi - I'm curious about something related to the best way to handle the ancillary data. Right now, L1 data orders can be accompanied by a set of corresponding ancillary data. Is it more efficient to download all of these ancillary data, move them into the proper directories, then run the l2gen process (in which case the ancillary data would be already on a local hard disk) or is it better to forget the bulk upfront download and let the getanc function of l2gen get the files? I ask because my attempts to
download the ancillary files using w.get seem to be randomly hanging/failing. If this step doesn't really make the whole L1 -> L2 processing more efficient, I'd like to avoid it.
I have the same questions, do we need these ancillary files to process from L1A to L1B, then from L1B to L2? I also like to skip, if not need.
My guess is that if w.get is having problems then the getanc function of l2gen will likely have them as well. In general, when dealing with unreliable processes it is better to break things down into simple steps and deal with the problems one-by-one. In your case there might be an exception if w.get is triggering some anti-DOS or traffic shaping mechanism. To combat that you could try to adjust the parameters for w.get. Are you running w.get once for each file in a list or passing a list to a single invocation of wget? If you use w.get for a single file then some of the anti-anti-DOS measures (--wait=seconds, --waitretry=seconds, and --random-wait) won't apply. Some large sites give priority to certain traffic classes under heavy load (and a site with lots of Windows PC's are expected to have significant spikes in spam/malware traffic), so you may have better results doing batch downloads during off-peak hours, using at/batch, etc. At large sites, traffic-shaping/anti-DOS rules may be applied to user workstations but not to servers, so if you can arrange to run batch downloads using a box on a subnet that is not subject to traffic-shaping/anti-DOS rules the problems may disappear. Such boxes are likely to have w.get installed, but not SeaDAS!