Slow Oceans FTP

By quinten.vanhell Date 2012-03-26 10:31

Hi all,

Since mid-February we are experiencing very slow connections to the oceans FTP (ftp://oceans.gsfc.nasa.gov/). As a result I have been unable to get near real-time data from my subscriptions.

Any ideas?

Best regards,

Quinten

By @chris Date 2012-03-27 20:34

Quinten,

Are you seeing slow download rates or delays in connecting to the server?

What program are you using to access oceans? A script or a commercial program?

How many files do you need to download?

We might be able to point you in the right direction with a little more information.

By quinten.vanhell Date 2012-03-29 08:24

Hi Chris,

Sorry for the lack of details. I used a combination of Firefox, Filezilla and Perl (LWP::UserAgent) scripts. The script gets the new files from three directories with cropped L2 data for rather small regions, so maybe 1-2 (small) images a day. The directory listing times out in Perl, but I could try to increase the time-out.

In Firefox getting the directory listing takes a long time, or doesn't work at all. In Filezilla the connection sometimes works and download speeds are acceptable (100Kb/s), but often the connection times out as well.

Quinten

By @chris Date 2012-03-30 20:22

Quinten,

There is a few second delay when logging into oceans. This is intentional and in place to help stop malicious scanners. Another issue you might see is if the server is quite busy or if there are a lot of files in a single directory then oceans might take a bit longer to respond. I would agree to increase the timeout in the perl script.

Once you have the file names use LWP::UserAgent or just wget to retrieve the files. Our server is on a multi gigabit link and will not limit the speed of your download. Speed will vary depending on the quality of the connection between our server and your client.

Hope this helps.

By treble Date 2012-05-22 22:56 Edited 2012-05-22 23:13

I am having problems with the FTP server oceans as well. Actually for quite a while. It doesn't always 'connect', but once it connects it is fast (like >700K/s). I have an automated script that downloads and processes the file, but it often fails to download scenes. Running the same command from the command prompt gives the same result. This is using wget on Centos 5.

Typical example:
wget -r --retry-connrefused --tries=20 --timeout=30 --retr-symlinks ftp://oceans.gsfc.nasa.gov:21/MODISA/XM/xxxx/xxxx/A2012142125500.L2_LAC.xxxx.hdf
--2012-05-22 15:40:50-- ftp://oceans.gsfc.nasa.gov/MODISA/XM/xxxx/xxxx/A2012142125500.L2_LAC.xxxx.hdf
=> `oceans.gsfc.nasa.gov/MODISA/XM/xxxx/xxxx/.listing'
Resolving oceans.gsfc.nasa.gov... 169.154.128.45
Connecting to oceans.gsfc.nasa.gov|169.154.128.45|:21... connected.
Logging in as anonymous ...
Error in server response, closing control connection.
Retrying.

And then it retries for another 19 times without success.... It 'hangs' after 'Logging in as anonymous' before it gets the error in server response.

Is there any preferred wget command?

By treble Date 2012-05-23 16:39

Replying to my own post here... but I have rewritten my scripts and now use a combination of curl and wget, retrieving not the symlink but the actual location of the file. So far it works well, but it may just be a coincidence as well and that the ftp server has a good day...

Getting file-listing first:
curl ftp://oceans.gsfc.nasa.gov/MODISA/XM/xxxx/xxxx/ > ftp-list.txt
This will contain a list of the scenes on the ftp-site including the link to the actual location of the file.
Using some IDL strsplit and strmatch command it is not too hard to extract the file location. Then use wget to retrieve it, like:
wget -r --retry-connrefused --tries=20 --timeout=30 ftp://oceans.gsfc.nasa.gov/MODISA/XM/xxxx/xxxx/../../../../mnt/ocdist2/MODISA/XM/xxxx/xxxx/A2011097202500.L2_LAC.xxxx.hdf
Note that I am now not trying to download:
ftp://oceans.gsfc.nasa.gov/MODISA/XM/xxxx/xxxx/A2011097202500.L2_LAC.xxxx.hdf
But:
ftp://oceans.gsfc.nasa.gov/MODISA/XM/xxxx/xxxx/../../../../mnt/ocdist2/MODISA/XM/xxxx/xxxx/A2011097202500.L2_LAC.xxxx.hdf

It looks like wget is a bit flakey with symbolic links...

By treble Date 2012-05-23 16:59

Hmmmmmm, celebrated too early. The ftp server is stuck again...

By @sean Date 2012-05-23 19:25

Try something like this:
wget "http://oceandata.sci.gsfc.nasa.gov/cgi/getfile/A2011097202500.L2_LAC.xxxx.hdf?h=oceans&p=/MODISA/XM/xxxx/xxxx"

By treble Date 2012-05-23 21:59

Wow, that works a lot better! Is it also possible to get a directory listing of my subscription through the oceandata.sci.gsfc.nasa.gov route? Thanks!

By @sean Date 2012-05-23 22:27 Edited 2012-05-23 22:33

It's on the todo list, unfortunately it's a looong list....

We have something in place for extracted data orders, which are staged on our distribution servers (a file called http_manifest.txt would reside in the order subdirectory).
However, nothing is in place for subscriptions...yet.

By treble Date 2012-05-23 22:34

OK, not a biggie. I am using ncftpls now, instead of wget, and that seems to work a bit more reliable.

I must say, going through http instead of ftp for downloading my scenes make a huge difference. It is much more responsive! I'm flying through my files now!

Thanks for your tips!