Not logged inOcean Color Forum
Up Topic Frequently Asked Questions / Data Access FAQ / Change to Subscription Data Access (locked)
- By SeanBailey Date 2013-06-19 13:18 Edited 2018-03-13 15:44
Greetings OBPG Data Users,

We would like to announce that we have a new method for distributing data files for the subscriptions
and will be phasing out the old FTP method as of August 1, 2013.

Previously files matching subscriptions were staged as zero-length files in subscription directories on our [url=ftp://oceans.gsfc.nasa.gov/subscriptions/]FTP site[/url]. 
This allowed users to use FTP to discover the file names they should download via our HTTP-getfile service.

The new method uses a new HTTP service that returns a list of URLs to the files in your subscription,
and its output can be piped directly into another wget command to download the files.  If you are a
curl user, it is a bit more complicated but still simpler than the FTP/HTTP combination.

The HTTP service is called with wget like this, where NNNN is replaced by your subscription ID:

wget --post-data="subID=NNNN&addurl=1&results_as_file=1" -O -  https://oceandata.sci.gsfc.nasa.gov/api/file_search

For example, to download a list of URLs to the files for the last 3 days for subscription 1003, use this command:

wget --post-data="subID=1003&addurl=1&results_as_file=1" -O -  https://oceandata.sci.gsfc.nasa.gov/api/file_search

To actually download the files for the above, pipe that command to another wget command:

wget --post-data="subID=1003&addurl=1&results_as_file=1" -O - https://oceandata.sci.gsfc.nasa.gov/api/file_search | wget -i -

Curl users can get the list of file URLs with this command:

curl -d "subID=1003&addurl=1&results_as_file=1"  https://oceandata.sci.gsfc.nasa.gov/api/file_search

Each file in the list can be downloaded with a small script like this:

for file in $(<curl cmd from above> | grep getfile)
do
    curl -L -O $file
done


The new HTTP method is available now, and as a convenience, we will be maintaining the old FTP
zero-length-file method until August 1, 2013 to allow users time to transition to the HTTP method.

If you have any questions or problems, please post them to the OBPG Satellite Data Access forum

Regards,
Sean
- By SeanBailey Date 2013-06-20 14:13
Dear OBPG Data Users,

There has been somewhat of a mixed reaction to yesterday's
announcement about the new distribution method for non-extracted data
subscriptions.  A number of users have expressed concern about the
amount of time given to make the transition from the
FTP/zero-length-file method to the HTTP-service method for file
discovery.

In response to that concern, we are prepared to move the cutoff date
for the FTP/zero-length-file method to August 1, 2013.

However, I would like to follow up the examples I provided in
yesterday's message with another one that might prove to be somewhat
simpler for those who want to continue to obtain a list of available
files in the subscription and feed that list to a separate download
method.

Most of you are using an FTP client to produce a directory listing for
your subscription.  Depending on the client and listing options, this
likely produces either a long listing such as this:

$ ncftpls -l ftp://oceans.gsfc.nasa.gov/subscriptions/1003/
-rw-r--r--    1 ftp      ftp             0 Jun 17 16:34 A2013168133500.L2_LAC_SST.bz2
-rw-r--r--    1 ftp      ftp             0 Jun 18 05:00 A2013169012000.L2_LAC_SST.bz2
-rw-r--r--    1 ftp      ftp             0 Jun 18 04:37 A2013169012500.L2_LAC_SST.bz2
-rw-r--r--    1 ftp      ftp             0 Jun 18 06:36 A2013169030000.L2_LAC_SST.bz2
-rw-r--r--    1 ftp      ftp             0 Jun 18 13:39 A2013169105500.L2_LAC_SST.bz2
-rw-r--r--    1 ftp      ftp             0 Jun 18 13:39 A2013169110000.L2_LAC_SST.bz2

or a short listing of only file names like this:

$ ncftpls ftp://oceans.gsfc.nasa.gov/subscriptions/1003/
A2013168133500.L2_LAC_SST.bz2
A2013169012000.L2_LAC_SST.bz2
A2013169012500.L2_LAC_SST.bz2
A2013169030000.L2_LAC_SST.bz2
A2013169105500.L2_LAC_SST.bz2
A2013169110000.L2_LAC_SST.bz2

The resulting list is then processed to determine which files need to
be downloaded, and for those that do, the HTTP-getfile URL is
constructed and passed to a program such as wget or curl.

All we are changing is this file-discovery method.  The examples I
provided in yesterday's message assumed that you would want to have
the HTTP-getfile URL given to you in the list of available files.  If
you wanted to continue to process each file separately to determine
which files to download, you would need to parse the file name from
the HTTP-getfile URLs, as in this wget example:

$ wget --post-data="subID=1003&addurl=1&results_as_file=1" -O - http://oceandata.sci.gsfc.nasa.gov/search/file_search.cgi 2> /dev/null
Your query generated 14 results, which are displayed below

http://oceandata.sci.gsfc.nasa.gov/cgi/getfile/A2013168003500.L2_LAC_SST.bz2
http://oceandata.sci.gsfc.nasa.govcgi/getfile/A2013168004000.L2_LAC_SST.bz2
http://oceandata.sci.gsfc.nasa.gov/cgi/getfile/A2013168021500.L2_LAC_SST.bz2
http://oceandata.sci.gsfc.nasa.gov/cgi/getfile/A2013168022000.L2_LAC_SST.bz2
http://oceandata.sci.gsfc.nasa.gov/cgi/getfile/A2013168035500.L2_LAC_SST.bz2
http://oceandata.sci.gsfc.nasa.gov/cgi/getfile/A2013168084000.L2_LAC_SST.bz2
http://oceandata.sci.gsfc.nasa.gov/cgi/getfile/A2013168101500.L2_LAC_SST.bz2
http://oceandata.sci.gsfc.nasa.gov/cgi/getfile/A2013168115500.L2_LAC_SST.bz2
http://oceandata.sci.gsfc.nasa.gov/cgi/getfile/A2013168133500.L2_LAC_SST.bz2
http://oceandata.sci.gsfc.nasa.gov/cgi/getfile/A2013169012000.L2_LAC_SST.bz2
http://oceandata.sci.gsfc.nasa.gov/cgi/getfile/A2013169012500.L2_LAC_SST.bz2
http://oceandata.sci.gsfc.nasa.gov/cgi/getfile/A2013169030000.L2_LAC_SST.bz2
http://oceandata.sci.gsfc.nasa.gov/cgi/getfile/A2013169105500.L2_LAC_SST.bz2
http://oceandata.sci.gsfc.nasa.gov/cgi/getfile/A2013169110000.L2_LAC_SST.bz2

However, you can choose to have only the file names returned by
omitting the addurl=1 parameter or setting its value to 0 like this:

$ wget --post-data="subID=1003&results_as_file=1" -O - http://oceandata.sci.gsfc.nasa.gov/search/file_search.cgi 2> /dev/null
Your query generated 14 results, which are displayed below

A2013168003500.L2_LAC_SST.bz2
A2013168004000.L2_LAC_SST.bz2
A2013168021500.L2_LAC_SST.bz2
A2013168022000.L2_LAC_SST.bz2
A2013168035500.L2_LAC_SST.bz2
A2013168084000.L2_LAC_SST.bz2
A2013168101500.L2_LAC_SST.bz2
A2013168115500.L2_LAC_SST.bz2
A2013168133500.L2_LAC_SST.bz2
A2013169012000.L2_LAC_SST.bz2
A2013169012500.L2_LAC_SST.bz2
A2013169030000.L2_LAC_SST.bz2
A2013169105500.L2_LAC_SST.bz2
A2013169110000.L2_LAC_SST.bz2

The above list of files can then be processed individually as though
they were parsed from an FTP listing.

It is our hope that this clarification coupled with the additional
month's time to transition will reduce any anxiety brought on by
yesterday's message.  Please continue to post feedback to the OBPG
Satellite Data Access forum

Regards,
john
Up Topic Frequently Asked Questions / Data Access FAQ / Change to Subscription Data Access (locked)

Powered by mwForum 2.29.7 © 1999-2015 Markus Wichitill