Not logged inOcean Color Forum
Forum Ocean Color Home Help Search Login
Up Topic Products and Algorithms / Satellite Data Access / getanc question (locked)
- By fields Date 2009-12-16 14:16
Is there a limit to the number of files I can grab in a period  of time?   I was going to loop through all my seawifs L2s, to get the ancillary on disk, before processing.  I usually get a couple before it hangs -

-----------------------
[fields@crash REPOR2009.1]$ getanc /raid/oceancolor/fields/seawifs/1997-11/S1997307200439.L1A_HMBR.reg -no-ice -verbose
Determining pass start and end times..

Input file : /raid/oceancolor/fields/seawifs/1997-11/S1997307200439.L1A_HMBR.reg
Start time : 1997307201000
End time   : 1997307201101

--16:13:15--  http://oceancolor.gsfc.nasa.gov/sdpscgi/public/display_ancillary_files.cgi?type=anc&start_time=1997307201000&stop_time=1997307201101&mission_letter=X&opt_flag=7
          => `S1997307200439.L1A_HMBR.reg.server'
Resolving oceancolor.gsfc.nasa.gov... 169.154.128.44
Connecting to oceancolor.gsfc.nasa.gov[169.154.128.44]:80... failed: Connection timed out.
Retrying.

--16:14:16--  http://oceancolor.gsfc.nasa.gov/sdpscgi/public/display_ancillary_files.cgi?type=anc&start_time=1997307201000&stop_time=1997307201101&mission_letter=X&opt_flag=7
(try: 2) => `S1997307200439.L1A_HMBR.reg.server'
Connecting to oceancolor.gsfc.nasa.gov[169.154.128.44]:80... failed: Connection timed out.
Giving up.

getanc: ERROR: The display_ancillary_files.cgi script encountered an
error and returned the following text:

-------------------------------
Once it hangs it seems to hang for other machines in our network.
- By sean Date 2009-12-17 09:35
Erik,

The is a three concurrent file download limit when accessing oceandata.sci.gsfc.nasa.gov,
but that isn't the problem here, as you weren't downloading files yet.  The error suggests a
network connection issue.  For some reason, you couldn't connect to our web server
(oceancolor.gsfc.nasa.gov NOT oceandata.sci.gsfc.nasa.gov).

We'll look into things on this end to see if there is an issue with our network, but as it seems
you were able to access the site later in the day, can I assume you were successful in
downloading the ancillary data you need?

Sean
- By fields Date 2009-12-17 10:26
Thanks for the reply Sean. I wasn't successful.  Later in the our sys admin did test to try different delays.  He wrote-

He was downloading the same file over and over, not using getanc. 

He wrote that "the evidence is stacking up to indicate that it's some sort of IP keyed limiting system on their end.  The counters seem to reset every hour. ..So my guess is that they expect <6 connections in 30 seconds."

I think he found 5 seconds between downloads worked.
- By sean Date 2009-12-17 13:42
Erik,

OK.  I've confirmed that there is a limit on connections per second to the web server,
primarily to prevent overly aggressive users/abusers.
 
I've made a minor modification to the anc search script to slow you down a bit.  Not
an ideal solution, but should prevent you from being stuck in the greylist just by
trying to get your work done.  It shouldn't be too terribly slow, though. 

Give it a try again.  If you trip up again, let us know.

Sean
- By fields Date 2009-12-17 14:27
Thanks Sean.

You did the modification on your side right?  Or was I suppose to modify my copy of getanc? 

It still hangs.    Should I put "sleeps" between my $download_cmd lines in getanc?
- By sean Date 2009-12-17 16:38
I put it in the script that gets called by getanc.  If it's still hanging, you may have to limit on your end.
You shouldn't need to modify getanc,  just loop through multiple calls to it slower.

Sean
- By Sayoob Date 2010-02-17 03:32 Edited 2010-02-17 04:03
Hii

What is getanc for...? While i tried to execute the automation script for processing MODIS L1A to L2 data it shows error
getanc: command not found

How can i fix it..?

Regards
Sayoob
- By mike Date 2010-02-17 11:02
getanc is a program that determines the ancillary data required for optimal Level-2 processing (meteorological, ozone, SST, NO2, sea ice). So in order to obtain the most accurate scientific products you need to run getanc to determine/download this ancillary data before processing Level-1 data to Level-2.

getanc is part of SeaDAS 6.0. So if you are using an earlier version of SeaDAS you must upgrade.

If you do have SeaDAS 6.0 installed and you get the "getanc: command not found" message then SeaDAS was not installed correctly. If this is the case, try the following:

1) change directory into your main SeaDAS 6.0 directory

2) if you have an IDL license execute "./config/seadas_setup" otherwise execute "./config/seadas_setup -em"   (two IDL windows should appear where you click "I Accept" and then "Okay".

3) make sure your shell init files have only one line that sources the seadas.env file (e.g. "source /home/seadas6/config/seadas.env"). If your shell is bash look in ~/.bashrc and ~/.bash_profile. If your shell is csh or tcsh look inside ~/.cshrc or ~/.tcshrc.

4) open a new terminal window and execute getanc to make sure it is working.
- By evelynors Date 2010-04-09 05:09
Hi,
Something wrong happend when I generated L2 data from L1A SeaWiFS data, I thought it must be something wrong related to the getanc.

After I saw this topic, I executed getanc. Just look details below. Obviously, it wasn't working.

seadas@seadasva:~$ getanc
Usage: getanc [L1A_or_L1B_file OR -start YYYYDDDHHMMSS] [-no-no2] [OTHER OPTIONS]

  L1A_or_L1B_file      Input L1A or L1B MODIS, SeaWiFS, OCTS, CZCS, or L1B MERIS file
  -start YYYYDDDHHMMSS Time of the first scanline (if used, no input file is required)
  -end YYYYDDDHHMMSS   Time of last scan (if unspecified and no input file, set to start+15min)
  -ancdir directory    Use a custom (non-SeaDAS) directory tree for ancillary files
  -curdir              Download ancillary files directly into current working directory
  -no-met              Do not search for meteorological ancillary data
  -no-ozo              Do not search for ozone ancillary data
  -no-sst              Do not search for SST ancillary data
  -no-no2              Do not search for NO2 ancillary data
  -no-ice              Do not search for sea ice ancillary data
  -no-download         Disable download of ancillary files not found on hard disk
  -verbose             Enable verbose server connection messages

NOTE: Currently NO2 climatological data is used for OBPG operational processing, so to match
      OBPG distributed data products, the SeaDAS default behaviour (triggered via the AUTO_NO2
      environment variable) disables NO2 searching. The -no-no2 option forces this as well.

This program queries an OBPG server and optionally downloads the optimal ancillary data
files for Level-1 to Level-2 processing. If an input file is specified the start and end
times are determined automatically, otherwise a start time must be provided by the user.
A text file (with the extension '.anc') is created containing l2gen parameters that can be
directly used as input to the l2gen program for optimal Level-1 to Level-2 processing, e.g.:

  l2gen ifile=<infile> ofile=<outfile> par=<the *.anc text file generated by this program>

EXIT STATUS = 0  : all optimal ancillary files exist and are present on the local disk
EXIT STATUS = 99 : an error was encountered and no .anc parameter text file was created
EXIT STATUS = 31 : no ancillary files currently exist corresponding to the start time and
                   therefore no .anc parameter text file was created

EXIT STATUS = 1-30 : bitwise value indicating one or more ancillary files are not optimal:

  bit 0 set = missing one or more MET files (if any files found, all 3 params still populated)
  bit 1 set = missing one or more OZONE files (if any files found, all 3 params still populated)
  bit 2 set = no SST file found
  bit 3 set = no NO2 file found
  bit 4 set = no ICE file found

e.g. EXIT STATUS=11 (2^0+2^1+2^3) means there are missing optimal MET, OZONE, and NO2 files

NOTE: If some but not all optimal MET or OZONE files are found, all three MET or OZONE
      parameters are populated with the found files and a warning message is displayed.

So, I followed these 4 steps, but, It didn't work. Could you help me?

Your reply will be appreciated!
Evelyn
- By sean Date 2010-04-09 07:57
Evelyn,

If you execute getanc on the command line without any arguments
it displays a usage statement.  This is expected and not a problem.
The first line summarizes how it is used.

     Usage: getanc [L1A_or_L1B_file OR -start YYYYDDDHHMMSS] [-no-no2] [OTHER OPTIONS]

What error messages, if any, were displayed when you processed the file?

Did you use the GUI or the command line?

Sean
- By evelynors Date 2010-04-09 21:42
Sean,
Thank you for your reply!
Actually&#65292; I used the GUI.
But, when I inputted my L1A file, a window just popped up and told me this:
A problem was encountered while attempting to determine the corresponding ancillary files. Please check your SeaDAS terminal window from more detail.
And, the terminal window:

Executing: $SEADAS/scripts/getanc /home/seadas/seadas6.0/Satellite Data/S2006209053105.L1A_GAC -no-no2
Usage: getanc [L1A_or_L1B_file OR -start YYYYDDDHHMMSS] [-no-no2] [OTHER OPTIONS]

  L1A_or_L1B_file      Input L1A or L1B MODIS, SeaWiFS, OCTS, CZCS, or L1B MERIS file
  -start YYYYDDDHHMMSS Time of the first scanline (if used, no input file is required)
  -end YYYYDDDHHMMSS   Time of last scan (if unspecified and no input file, set to start+15min)
  -ancdir directory    Use a custom (non-SeaDAS) directory tree for ancillary files
  -curdir              Download ancillary files directly into current working directory
  -no-met              Do not search for meteorological ancillary data
  -no-ozo              Do not search for ozone ancillary data
  -no-sst              Do not search for SST ancillary data
  -no-no2              Do not search for NO2 ancillary data
  -no-ice              Do not search for sea ice ancillary data
  -no-download         Disable download of ancillary files not found on hard disk
  -verbose             Enable verbose server connection messages

NOTE: Currently NO2 climatological data is used for OBPG operational processing, so to match
      OBPG distributed data products, the SeaDAS default behaviour (triggered via the AUTO_NO2
      environment variable) disables NO2 searching. The -no-no2 option forces this as well.

This program queries an OBPG server and optionally downloads the optimal ancillary data
files for Level-1 to Level-2 processing. If an input file is specified the start and end
times are determined automatically, otherwise a start time must be provided by the user.
A text file (with the extension '.anc') is created containing l2gen parameters that can be
directly used as input to the l2gen program for optimal Level-1 to Level-2 processing, e.g.:

  l2gen ifile=<infile> ofile=<outfile> par=<the *.anc text file generated by this program>

EXIT STATUS = 0  : all optimal ancillary files exist and are present on the local disk
EXIT STATUS = 99 : an error was encountered and no .anc parameter text file was created
EXIT STATUS = 31 : no ancillary files currently exist corresponding to the start time and
                   therefore no .anc parameter text file was created

EXIT STATUS = 1-30 : bitwise value indicating one or more ancillary files are not optimal:

  bit 0 set = missing one or more MET files (if any files found, all 3 params still populated)
  bit 1 set = missing one or more OZONE files (if any files found, all 3 params still populated)
  bit 2 set = no SST file found
  bit 3 set = no NO2 file found
  bit 4 set = no ICE file found

e.g. EXIT STATUS=11 (2^0+2^1+2^3) means there are missing optimal MET, OZONE, and NO2 files

NOTE: If some but not all optimal MET or OZONE files are found, all three MET or OZONE
      parameters are populated with the found files and a warning message is displayed.

I've already mentioned this on the other post, someone told me to update my seaDAS 6.0 to 6.1 version. But, I think it might be problems related to getanc script, I don't know. What does that "no .anc parameter text file was created" mean?
By the way, when I executed my steps on SeaDAS 4.4 before, it was easy and there was no such error that stopped me from generating L2. Could you help me? Sean. I'd really appreciate your help.

Best regards
Evelyn
- By sean Date 2010-04-10 09:55
Evelyn,

I suspect the problem is that your path to the satellite file includes a space:

    /home/seadas/seadas6.0/Satellite Data/S2006209053105.L1A_GAC
                                  ^


Since the path isn't quoted or the space escaped, the program interprets:

     /home/seadas/seadas6.0/Satellite

as the first argument, which is incorrect.  The simplest solution is to
rename the directory Satellite Data to SatelliteData

Sean
- By evelynors Date 2010-04-10 21:56
Hi sean,

Thank you for your help yesterday. it worked! I finally input the L1A file.
But, I don't know why those ancillary data should be download directly? look:

Input file : /home/seadas/seadas6.0/SatelliteData/S2006209053105.L1A_GAC
Start time : 2006209053104
End time   : 2006209061440

Downloading N200620800_O3_TOMSOMI_24h.hdf
Downloading N2006209_SST_OIV2AVAM_24h.nc
Downloading N200620900_SEAICE_NSIDC_24h.hdf

I've already downloaded all of them to my local disk, the SatelliteData file, why it just redownloaded again? Is that mean I don't need to download any ancillary data from this website to my windows any more? Just let this input process to download and find the optimal?

And when all the optimal ancillary data were determinged and found, it seems that I still need to choose the "parameter files", which I don't know how. If I ignore this parameter file, the error happens like this:
Process SeaWiFS File
Created L2GEN parameter file:
/home/seadas/seadas6.0/SatelliteData/S2006209053105.L2_GAC.par
UNIX command:
/bin/bash -c "l2gen par=/home/seadas/seadas6.0/SatelliteData/S2006209053105.L2_GAC.par"
Input file /home/seadas/seadas6.0/SatelliteData/S2006209053105.L1A_GAC is SeaWiFS Level-1A GAC.
Loading default parameters for SeaWiFS from /home/seadas/seadas6.0/data/seawifs/msl12_defaults.par

Invalid argument "owtfile=$OCDATAROOT/seawifs/class/seawifs_owt_stats.hdf"
-E- msl12_input.c Line 428: Error reading input parameters.
-E- l2gen: Error parsing input parameters.

exit_status=            1

Thanks again!
Best regards,
Evelyn
- By sean Date 2010-04-12 08:51
Evelyn,

The par file is created by the GUI.  IF you have one you've already made that
you would like to include, the GUI gives you the ability to do so.  You do
not need to enter any par file, it's optional.

The error you're seeing is not related to this, though it is a parameter file
issue.  It seems as though you have a mix of versions 6.0 and 6.1.  The
l2gen version seems to be from 6.0, but the SeaWiFS default parameter file
is from 6.1.  Make sure that you use consistent bin and data directories.

Regards,
Sean
- By evelynors Date 2010-04-12 22:36
Hi sean,

Thanks for your patience and help, I am now totally confused and there is nobody else except people in this forum that I might ask for help. Please forgive my ignorance.

I know there is a par file: S2006210043252.L2_GAC.par, which was generated after "run" and is right at my directory of the L1A data. You mean I don't need to choose any other parameter file to fill the blank because it's optional.
How can I change this SeaWiFS default parameter file to 6.0 version? I tried to figure it out but I don't know how.

By the way, another "stupid" question. Every time I put in L1 data in the l2gen, it will download ancillary data from this website automatically.Even if I put ancillary data in my disk, it will still search for downloading to the directory:home/seadas/seadas6.0/var/anc/yyyy/ddd. Does that mean I don't need to download any ancillary data by my own?

2 quesions, Thank you for your time again, really.

Best regards,
Evelyn
- By sean Date 2010-04-13 08:31
Evelyn,

Each mission has a default parameter file - you never need to touch these.
To modify the L2 processing, using the GUI, you can change every
parameter to suite your needs.  The are several tabs within the L2 processing
GUI (subset/Subsample, Processing Options, Flag Thresholds, GIOP options)
SeaDAS will create a new parameter file with your changes which will be
used for the processing.  If you wish to replicate this numerous times you can
save the parameter file that SeaDAS creates and use it for subsequent files - saving
you the need to manually reselect the options again.  You may need to manually
edit this parameter file to remove the file specific items (ifile, ofile, MET, OZONE, etc).

You do NOT need to pre-download the ancillary data files.  SeaDAS will
do it for you when processing is initiated if it does not find the required files
in $SEADAS/var/anc/yyyy/ddd.  However, you can download them first,
placing them in the appropriate yyyy/ddd directory of the $SEADAS/var/anc
tree, and SeaDAS will detect them and not re-download them.

Sean
- By evelynors Date 2010-04-13 11:13
Thank you, sean

You have given me quite a lot help. I really appreciate that.

Best regards,
Evelyn
Up Topic Products and Algorithms / Satellite Data Access / getanc question (locked)



Responsible NASA Official: Gene C. Feldman
Curator: OceanColor Webmaster
Authorized by: Gene C. Feldman
Updated: 03 July 2013
Privacy Policy and Important Notices NASA logo