additional help on http bulk data downloads

By vlance Date 2010-09-28 18:29

Hello,
I am seeking more help or advice about "bulk data downloads" than covered on the the FAQ page:

I have retrieved a list of MODIS & SeaWIFS filenames for covering data for my desired time frame AND spatial coordinates.. This was generated via the web search utility and saved (the resulting list of filenames) as a .txt file for now.

I could accomplish a single download by entering the proper http:// etc address directly to my browser.
By following the instructions on the FAQ, I was also successful using the "curl" command - this worked fine for both MODIS and for SeaWIFS which required the user:password.
So far so good - however, I have hundreds to thousands of file names, so I would like to automate my data download process.

The examples on the FAQ for using the file search utility don't appear to allow for a defined location. As I already accomplished the "search" on the web quite easily - I suspect I should be able to automate the "curl" command such that it "looks" at the first line in my txt file and fills that into the http address, downloads, saves the data file and looks back at my txt at line 2 for the next file. Does anyone have examples for doing this?

(By the way, I have the MODIS filenames and the SeaWIFS filenames in separate .txt files so I don't need to complicate matters by adding " IF filename has starts with "A" THEN no password, if S then password" stuff, but i'm sure savvy people can do that easily).

Thanks very much for any help.
Veronica

By @norman Date 2010-09-28 20:15

Veronica,

How about something like this? (My example assumes a C-shell.)
foreach f ( `cat list.txt|xargs` ) curl -O http://oceandata.sci.gsfc.nasa.gov/cgi/getfile/$f.bz2 end
Norman

By @sean Date 2010-09-28 20:19

Veronica,

No, the file search utility doesn't do geographic searches - that's what the L1/L2 browser is best at.

Since you can run curl on one file, all you need to do is loop that process for multiple files.
Here's an example of how using a simple bash script (remove the extraneous dots):

#!/b.in/b.ash
for file in $(cat a.list);
do
curl -L -O http://oceandata.sci.gsfc.nasa.gov/cgi/getfile/$file.bz2;
done;

(note: a.list is a text file containing a list of files, one per line, as can be obtained from the L1/L2 browser)

You can use the same script for both SeaWiFS and MODIS, nothing prevents you from downloading MODIS data
while connecting as an authorized SeaWiFS user... just change 'cgi' to 'restrict' in the URL and enter your login info.

Regards,
Sean

By vlance Date 2010-10-04 19:58

Thanks very much, Sean and Norman.
After all operator errors were corrected (ugh!), Sean's "for file in" version worked for me on my trial list containing 2 modis filenames.
V-

By vlance Date 2011-02-23 22:10

Hello again,

So, now I am trying to skip a step or two and harness the power of computer programming.....

I have created a text file which contains the list of the year and day of year for which I would like to retrieve satellite files - but I want to get any and all of the timed L1A files for that day...
For example, my text file for modis looks like this:

A2002129
A2002133
A2004163
etc....

I tried using the bash commands from the post above by Sean, inserting wildcards to fill in for the time.
The command seems to work, but is looking for the literal file with the asterisk in the name...
I get back a list of "empty" files called literally:
A2002129*.L1A_LAC.bz2 , etc.
Can anyone help with the syntax to make the wildcards work?

Here is one of my cleanest failed attempts (where a.list in this case is now the text file with the "incomplete" satellite file names:
--------------------
for file in $(cat a.list);
do
curl -L -O http://oceandata.sci.gsfc.nasa.gov/cgi/getfile/$file*.L1A_LAC.bz2;
done;
------------------

the bash command advice i found for wildcards was never combined with a curl instruction/example... maybe curl needs to be more "literal"??? Or nested?
I found something called "eval" that might be useful, but i couldn't follow it for my purposes.

Thanks for any advice on this seemingly simple (but elusive for me) problem.
V-

By dmoonan Date 2011-02-23 22:40

Hello Veronica,

I see what you're trying to do, but the curl command requires the exact file name, since you are using cgi/getfile as the url reference. That is a pointer to a script ("getfile") which is going to need the full file name, and will not accept wild cards such as "*". Hope this helps.

Best Regards,
Daniel