Search and Download Methods

The OB.DAAC is a member of NASA's Earth Observing System Data and Information System (EOSDIS) Distributed Active Archive Centers (DAACs). As a member, a change to our data acess policy was necessary. Searching for OB.DAAC data will continue as-is with no additional requriements of the user. However, we now require users to login to the OceanColor Web's data access points using their Earthdata Login credentials in order to download any products. Although this extra step has been imposed on download operations, OB.DAAC data remains free and open to the public.

Search for files via Command Line Interface (CLI)

OB.DAAC has a file search utility that is accessible through command line interface. An API for file search is also available. See section below titled Downlaod via Web. User registration is not required to search for files.

Available options:

  • sensor: mission name.
    • valid options: aquarius, seawifs, aqua, terra, meris, octs, czcs, hico, viirs (for snpp), viirsj1, s3olci (for sentinel-3a), s3bolci
  • sdate - start date for a search (format YYYY-MM-DD)
  • edate - end date for a search (format YYYY-MM-DD)
  • psdate - file processing start date for a search (format YYYY-MM-DD)
  • pedate - file processing end date for a search (format YYYY-MM-DD)
  • dtype - data type (i.e. level).
    • valid options:
    • L0, L1, L2, L3b (for binned data), L3m (for mapped data), MET (for ancillary data), misc (for sundry products)
  • addurl - include full url in search result (boolean, 1=yes, 0=no)
  • results_as_file - return results as a test file listing (boolean, 1=yes, 0=no, thus returns and HTML page)
  • search - text string search
  • subID - subscription ID to search
  • subType - subsciption type, valid options are:
    • valid options:
    • 1 (non-extracted, default), 2 (extracted)
  • std_only - restrict results to standard products (i.e. ignore extracts, regional processings, etc.; boolean)
  • cksum - return a checksum file for search results (boolean; sha1sums except for Aquarius soil moisture products which are md5sums; forces results_as_file; ignores addurl)
  • format - file is returned in format specified.
    • valid options:
    • txt, json, html

Examples

  1. Use the file_search utility to find and download OCTS daily L3 binned chlorophyll data from November 1, 1996 through December 31, 1999

    Using Wget

    wget -q --post-data="sensor=octs&sdate=1996-11-01&edate=1997-01-01&dtype=L3b&addurl=1&results_as_file=1&search=*DAY_CHL*" -O - https://oceandata.sci.gsfc.nasa.gov/api/file_search

    Using cURL

    curl -d "sensor=octs&sdate=1996-11-01&edate=1997-01-01&dtype=L3b&addurl=1&results_as_file=1&search=*DAY_CHL*" https://oceandata.sci.gsfc.nasa.gov/api/file_search |grep getfile | cut -d "'" -f 2 | head -1 | xargs -n 1 curl -LJO -n -c ~/.urs_cookies -b ~/.urs_cookies

View search queries for finding OB.DAAC data in the Earthdata Search Tool

Download files via CLI

CAVEATS

  1. Examples are provided for informational purposes only.
  2. No product endorsement is implied.
  3. Results may vary based on the version of software installed on your machine.
  4. Examples given are not an exhaustive description of possibilities.

Python

The following is an example of how to use python to access data. Equivalent methods exist in the SeaDAS distributed code and can be used if SeaDAS is already installed under $OCSSWROOT/scripts/ProcUtils.py.

* The line terminators in the script may need to be changed to coorespond to the system from which you execute the script.


Additional examples for accessing data using Python can be found on the Earthdata Wiki.

Wget

Examples:

RECOMMENDED METHOD: Configure your username and password for authentication using a .netrc file.

echo "machine urs.earthdata.nasa.gov login USERNAME password PASSWD" > ~/.netrc ; > ~/.urs_cookies
chmod  0600 ~/.netrc

where USERNAME and PASSWD are your Earthdata Login credentials.

  1. Retrieve a single file using cookies to pass credentials:
    wget --load-cookies ~/.urs_cookies --save-cookies ~/.urs_cookies --auth-no-challenge=on --content-disposition https://oceandata.sci.gsfc.nasa.gov/ob/getfile/T2017004001500.L1A_LAC.bz2

    * Depending on the version of wget installed, the option --auth-no-challenge=on may be required.
    * Some users of wget may have issues with IPv6 and may need to include -4 in the wget call to force the use of IPv4.

  2. Retrieve a listing of MODIS-Aqua nighttime SST for Jan 5, 2006 using username and password only:
    wget -q -O - https://oceandata.sci.gsfc.nasa.gov/MODIS-Aqua/L2/2006/005/ |grep SST4| wget --user=USERNAME --ask-password --auth-no-challenge=on --base https://oceandata.sci.gsfc.nasa.gov/ -N --wait=0.5 --random-wait --force-html -i -

    * Depending on the version of wget installed, the option --auth-no-challenge=on may be required.
    * Some users of wget may have issues with IPv6 and may need to include -4 in the wget call to force the use of IPv4.

  3. Retrieve a single file by interactively passing username and password:
    wget --user=USERNAME --ask-password --auth-no-challenge=on https://oceandata.sci.gsfc.nasa.gov/ob/getfile/T2017004001500.L1A_LAC.bz2

    * Depending on the version of wget installed, the option --auth-no-challenge=on may be required.
    * Some users of wget may have issues with IPv6 and may need to include -4 in the wget call to force the use of IPv4.

Additional Options:

  • --timeout=10 - sets timeout to 10 seconds (by default wget will retry after timeout)
  • --wait=0.5 - tells wget to pause for 0.5 seconds between attempts
  • --random-wait - causes the time between requests to vary between 0.5 and 1.5 * wait seconds, where wait was specified using the --wait option
  • -N, --timestamping - prevents wget from downloading files already retrieved if a local copy exists and the remote copy is not newer

cURL

Examples:

Unlike wget, cURL has no method for downloading a list of URLs (although it can download multiple URLs on the command line). However, a shell or script (perl, python, etc) loop can easily be written to preform this task.

RECOMMENDED METHOD: Configure your username and password for authentication using a .netrc file.

echo "machine urs.earthdata.nasa.gov login USERNAME password PASSWD" > ~/.netrc ; > ~/.urs_cookies
chmod 0600 ~/.netrc

where USERNAME and PASSWD are your Earthdata Login credentials.

  1. Retrieve a single file using a cookies to pass credentials. This example retrieves a level-1a local area coverage data file for MODIS-Terra for Jan 4, 2017:
    curl -O -b ~/.urs_cookies -c ~/.urs_cookies -L -n https://oceandata.sci.gsfc.nasa.gov/ob/getfile/T2017004001500.L1A_LAC.bz2
  2. Retrieve a file list using cookies. This example returns MODIS L2 files for 2006 day 005 (Jan 5th, 2006):
    curl https://oceandata.sci.gsfc.nasa.gov/MODIS-Aqua/L2/2006/005/ | grep getfile | cut -d "'" -f 2 | head -1 | xargs -n 1 curl -LJO -n -c ~/.urs_cookies -b ~/.urs_cookies

Still having trouble? Try these resources:

Download via Web

OceanColor HTTPS

Go to the Direct Data Download page to access files over HTTPS.

File Search API

Use the File Search API to perform data searches and return results to the screen. File search can also be preformed via command line (See section above).

Earthdata Search Tool

A significant amount of our data are now hosted in the EOSDIS Earthdata Search Tool. If you have not registered with Earthdata Login, please follow these instructions to make an account. Users must use login with this account before any OB.DAAC data may be downloaded. Below are some examples of useful queries.

  1. Find collections from the data provider (OB.DAAC) that contain granules:
    https://cmr.earthdata.nasa.gov/search/collections.umm_json?provider=OB_DAAC&has_granules=true&page_size=100
    
    Get page 2,3 of the data:
    https://cmr.earthdata.nasa.gov/search/collections.umm_json?provider=OB_DAAC&has_granules=true&page_size=100&page_num=2
    -and-
    https://cmr.earthdata.nasa.gov/search/collections.umm_json?provider=OB_DAAC&has_granules=true&page_size=100&page_num=3
    
    With product level:
    https://cmr.earthdata.nasa.gov/search/collections.umm_json?provider=OB_DAAC&has_granules=true&processing_level_id=2

  2. Find granules based on data provider and short_name:
    https://cmr.earthdata.nasa.gov/search/granules.umm_json?page_size=10&sort_key=short_name&sort_key=start_date&short_name=MODISA_L3b_SST&provider=OB_DAAC
    
    There are other options like 'date', and 'geospatial' that can be sent, as well. 
    Temporal search:
    https://cmr.earthdata.nasa.gov/search/granules.umm_json?page_size=20&sort_key=short_name&sort_key=start_date&short_name=MODISA_L3b_SST&provider=OB_DAAC&temporal=2020-01-03,2020-01-10
    
    Bounding Box:
    https://cmr.earthdata.nasa.gov/search/granules.umm_json?page_size=20&sort_key=short_name&sort_key=start_date&short_name=MODISA_L3b_SST&provider=OB_DAAC&bounding_box=-10,-5,10,5&temporal=2020-01-03,2020-01-10
    
    Polygon:
    https://cmr.earthdata.nasa.gov/search/granules.umm_json?page_size=20&sort_key=short_name&sort_key=start_date&short_name=MODISA_L3b_SST&provider=OB_DAAC&polygon=10,10,30,10,30,20,10,20,10,10&temporal=2020-01-03,2020-01-10
    
         

More questions? View Earthdata Search FAQs