Search and Download Methods

The OB.DAAC is a member of NASA's Earth Observing System Data and Information System (EOSDIS) Distributed Active Archive Centers (DAACs). As a member, a change to our data acess policy was necessary. Searching for OB.DAAC data will continue as-is with no additional requirements of the user. However, we now require users to login to the OceanColor Web's data access points using their Earthdata Login credentials in order to download any products. Although this extra step has been imposed on download operations, OB.DAAC data remains free and open to the public.

Continue below for search methods or jump to sections on: download methods, retrieving orders, or setting up subscriptions.

Search Methods

OB.DAAC has a file search utility that is accessible through command line interface (CLI). An API for file search is also available. User registration is not required to search for files.

Available options:

  • sensor: mission name.
    • valid options: aquarius, seawifs, aqua, terra, meris, octs, czcs, hico, viirs (for snpp), viirsj1, s3olci (for sentinel-3a), s3bolci
  • sdate - start date for a search (format YYYY-MM-DD)
  • edate - end date for a search (format YYYY-MM-DD)
  • psdate - file processing start date for a search (format YYYY-MM-DD)
  • pedate - file processing end date for a search (format YYYY-MM-DD)
  • dtype - data type (i.e. level).
    • valid options:
    • L0, L1, L2, L3b (for binned data), L3m (for mapped data), MET (for ancillary data), misc (for sundry products)
  • addurl - include full url in search result (boolean, 1=yes, 0=no)
  • results_as_file - return results as a test file listing (boolean, 1=yes, 0=no, thus returns and HTML page)
  • search - text string search
  • subID - subscription ID to search
  • subType - subscription type, valid options are:
    • valid options:
    • 1 (non-extracted, default), 2 (extracted)
  • std_only - restrict results to standard products (i.e. ignore extracts, regional processings, etc.; boolean)
  • cksum - return a checksum file for search results (boolean; sha1sums except for Aquarius soil moisture products which are md5sums; forces results_as_file; ignores addurl)
  • format - file is returned in format specified.
    • valid options:
    • txt, json, html

Command line interface

  1. Use the file_search utility to find and download OCTS daily L3 binned chlorophyll data from November 1, 1996 through December 31, 1999

    Wget:

    wget -q --post-data="sensor=octs&sdate=1996-11-01&edate=1997-01-01&dtype=L3b&addurl=1&results_as_file=1&search=*DAY_CHL*" -O - https://oceandata.sci.gsfc.nasa.gov/api/file_search

    cURL:

    curl -d "sensor=octs&am;sdate=1996-11-01&edate=1997-01-01&dtype=L3b&addurl=1&results_as_file=1&search=*DAY_CHL*" https://oceandata.sci.gsfc.nasa.gov/api/file_search |grep getfile | cut -d "'" -f 2 | head -1 | xargs -n 1 curl -LJO -n -c ~/.urs_cookies -b ~/.urs_cookies

OB.DAAC File Search API

Use the File Search API to perform data searches and return results to the screen. File search can also be preformed via command line (See section above).

Earthdata Search Tool

A significant amount of our data are now hosted in the EOSDIS Earthdata Search Tool. If you have not registered with Earthdata Login, please follow these instructions to make an account. Users must use log into their Earthdata account before any OB.DAAC data may be downloaded. Useful queries follow:

  1. Find collections from the data provider (OB.DAAC) that contain granules:
    https://cmr.earthdata.nasa.gov/search/collections.umm_json?provider=OB_DAAC&has_granules=true&page_size=100
    Get page 2, 3 of the data:
    https://cmr.earthdata.nasa.gov/search/collections.umm_json?provider=OB_DAAC&has_granules=true&page_size=100&page_num=2
    
    -and-
    
    https://cmr.earthdata.nasa.gov/search/collections.umm_json?provider=OB_DAAC&has_granules=true&page_size=100&page_num=3
    With product level:
    https://cmr.earthdata.nasa.gov/search/collections.umm_json?provider=OB_DAAC&has_granules=true&processing_level_id=2
  2. Find granules based on data provider and short_name:
    https://cmr.earthdata.nasa.gov/search/granules.umm_json?page_size=10&sort_key=short_name&sort_key=start_date&short_name=MODISA_L3b_SST&provider=OB_DAAC

    There are other options like 'date', and 'geospatial' that can be sent, as well.

    Temporal search:

    https://cmr.earthdata.nasa.gov/search/granules.umm_json?page_size=20&sort_key=short_name&sort_key=start_date&short_name=MODISA_L3b_SST&provider=OB_DAAC&temporal=2020-01-03,2020-01-10
    Bounding Box:
    https://cmr.earthdata.nasa.gov/search/granules.umm_json?page_size=20&sort_key=short_name&sort_key=start_date&short_name=MODISA_L3b_SST&provider=OB_DAAC&bounding_box=-10,-5,10,5&temporal=2020-01-03,2020-01-10
    Polygon:
    https://cmr.earthdata.nasa.gov/search/granules.umm_json?page_size=20&sort_key=short_name&sort_key=start_date&short_name=MODISA_L3b_SST&provider=OB_DAAC&polygon=10,10,30,10,30,20,10,20,10,10&temporal=2020-01-03,2020-01-10

More questions? View Earthdata Search FAQs

Download Methods

CAVEATS

  • Examples are provided for informational purposes only.
  • No product endorsement is implied.
  • Results may vary based on the version of software installed on your machine.
  • Examples given are not an exhaustive description of possibilities.
  1. Create a .netrc file.

    RECOMMENDED METHOD: Configure your username and password for authentication using a .netrc file. If you experience errors or redirects when using a cookie file, delete any existing cookie files and generate a new one for your current session.

    echo "machine urs.earthdata.nasa.gov login USERNAME password PASSWD" > ~/.netrc ; > ~/.urs_cookies
    chmod  0600 ~/.netrc

    where USERNAME and PASSWD are your Earthdata Login credentials.

  2. Choose a download method.
    Use a python script

    The following is an example of how to use python to access data. Equivalent methods exist in the SeaDAS distributed code and can be used if SeaDAS is already installed under $OCSSWROOT/scripts/ProcUtils.py.

    usage:

    obdaac_download [-h] [-v] [--filelist FILELIST] [--http_manifest HTTP_MANIFEST] [--odir ODIR] [--uncompress] [--force] [filename]
    
    • - positional arguments:

      • filename - name of the file (or the URL of the file) to retrieve
    • - optional arguments:

      • -h, --help - show this help message and exit
      • -v, --verbose - print status messages
      • --filelist FILELIST - file containing list of filenames to retrieve, one per line
      • --http_manifest HTTP_MANIFEST - URL to http_manifest file for OB.DAAC data order
      • --odir ODIR - full path to desired output directory; defaults to current working directory.
      • --uncompress - uncompress the retrieved files (if compressed)
      • --force - force download even if file already exists locally

      Provide one of either filename, --filelist or --http_manifest. NOTE: For authentication, a valid .netrc file in the user home ($HOME) is required, e.g.: machine urs.earthdata.nasa.gov login USERNAME password PASSWD.

    Download

    * The line terminators in the script may need to be changed to match those accepted by your operating system.

    Additional python examples can be found on the Earthdata Wiki.

    Use Wget

    Install | Learn

    Examples:

    1. Retrieve a single file using cookies to pass credentials:
      wget --load-cookies ~/.urs_cookies --save-cookies ~/.urs_cookies --auth-no-challenge=on --content-disposition https://oceandata.sci.gsfc.nasa.gov/ob/getfile/T2017004001500.L1A_LAC.bz2

      * Depending on the version of wget installed, the option --auth-no-challenge=on may be required.
      * Some users of wget may have issues with IPv6 and may need to include -4 in the wget call to force the use of IPv4.

    2. Retrieve a listing of MODIS-Aqua nighttime SST for Jan 5, 2006 using username and password only:
      wget -q -O - https://oceandata.sci.gsfc.nasa.gov/MODIS-Aqua/L2/2006/005/ |grep SST4| wget --user=USERNAME --ask-password --auth-no-challenge=on --base https://oceandata.sci.gsfc.nasa.gov/ -N --wait=0.5 --random-wait --force-html -i -

      * Depending on the version of wget installed, the option --auth-no-challenge=on may be required.
      * Some users of wget may have issues with IPv6 and may need to include -4 in the wget call to force the use of IPv4.

    3. Retrieve a single file by interactively passing username and password:
      wget --user=USERNAME --ask-password --auth-no-challenge=on https://oceandata.sci.gsfc.nasa.gov/ob/getfile/T2017004001500.L1A_LAC.bz2

      * Depending on the version of wget installed, the option --auth-no-challenge=on may be required.
      * Some users of wget may have issues with IPv6 and may need to include -4 in the wget call to force the use of IPv4.

    Additional Options:

    • --timeout=10 - sets timeout to 10 seconds (by default wget will retry after timeout)
    • --wait=0.5 - tells wget to pause for 0.5 seconds between attempts
    • --random-wait - causes the time between requests to vary between 0.5 and 1.5 * wait seconds, where wait was specified using the --wait option
    • -N, --timestamping - prevents wget from downloading files already retrieved if a local copy exists and the remote copy is not newer


    Use cURL

    Install | Learn

    Unlike wget, cURL has no method for downloading a list of URLs (although it can download multiple URLs on the command line). However, a shell or script (perl, python, etc) loop can easily be written to preform this task. Examples of use cases are below.

    1. Retrieve a single file using a cookies to pass credentials. This example retrieves a level-1a local area coverage data file for MODIS-Terra for Jan 4, 2017:
      curl -O -b ~/.urs_cookies -c ~/.urs_cookies -L -n https://oceandata.sci.gsfc.nasa.gov/ob/getfile/T2017004001500.L1A_LAC.bz2
    2. Retrieve a file list using cookies. This example returns MODIS L2 files for 2006 day 005 (Jan 5th, 2006):
      curl https://oceandata.sci.gsfc.nasa.gov/MODIS-Aqua/L2/2006/005/ | grep getfile | cut -d "'" -f 2 | head -1 | xargs -n 1 curl -LJO -n -c ~/.urs_cookies -b ~/.urs_cookies

Still having trouble? Try these resources:

Orders

To download the files for an order, you will first need to download the manifest file. The URL for the manifest file will be included in the email that is sent to the registered email address after an order is staged.

First download the http_manifest.txt file. This file contains URLs for each file in your order.

https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/http_manifest.txt?h=HOST&p=ORDER

where HOST is the name of the server hosting your file and ORDER is your order number. You can then use a program like Wget or cURL to download the files in your manifest file. NOTE: The OB.DAAC requires users to authenticate before downloading data. Please make sure that you have created a valid .netrc file in your home direcotry before attempting to execute the examples below.

wget --load-cookies ~/.urs_cookies --save-cookies ~/.urs_cookies --auth-no-challenge=on --no-check-certificate --content-disposition -i manifest-file-name

You can also pipe wget commands together to initiate the download:

wget -O - 'manifest-url-from-above' | wget --load-cookies ~/.urs_cookies --save-cookies ~/.urs_cookies --auth-no-challenge=on --no-check-certificate --content-disposition -i -

You an also use the python script mentioned above to download the manifest file by passing the filename or manifest URL to the script

obdaac_download [-v] [--http_manifest MANIFEST_URL_FROM_EMAIL] [--odir ODIR]

Subscriptions

To compile a list of URLs for your subscription and then download the data, use wget. NOTE: The OB.DAAC requires users to authenticate before downloading data. Please make sure that you have created a valid .netrc file in your home direcotry before attempting to execute the following example.

wget "https://oceandata.sci.gsfc.nasa.gov/api/file_search?subID=SUBSCR&subType=TYPE&format=txt&addurl=1" -O - | wget --load-cookies ~/.urs_cookies --save-cookies ~/.urs_cookies --auth-no-challenge=on --no-check-certificate --content-disposition -i -

where SUBSCR is your subscription request ID and TYPE is the subscription type. Valid options for TYPE are 1 (for non-extracted) or 2 (for extracted).

If you would like to manage your subscription(s), you may do so by visiting the OBPG Subscription Summary page.

If you have questions or experience problems, please direct them to our project forum, but please read the FAQ board first to see if a solution to your issue has already been published in one of them.