Ocean Color Forum - Not logged in
Since mid-February we are experiencing very slow connections to the oceans FTP (ftp://oceans.gsfc.nasa.gov/
). As a result I have been unable to get near real-time data from my subscriptions.
Are you seeing slow download rates or delays in connecting to the server?
What program are you using to access oceans? A script or a commercial program?
How many files do you need to download?
We might be able to point you in the right direction with a little more information.
Sorry for the lack of details. I used a combination of Firefox, Filezilla and Perl (LWP::UserAgent) scripts. The script gets the new files from three directories with cropped L2 data for rather small regions, so maybe 1-2 (small) images a day. The directory listing times out in Perl, but I could try to increase the time-out.
In Firefox getting the directory listing takes a long time, or doesn't work at all. In Filezilla the connection sometimes works and download speeds are acceptable (100Kb/s), but often the connection times out as well.
There is a few second delay when logging into oceans. This is intentional and in place to help stop malicious scanners. Another issue you might see is if the server is quite busy or if there are a lot of files in a single directory then oceans might take a bit longer to respond. I would agree to increase the timeout in the perl script.
Once you have the file names use LWP::UserAgent or just wget to retrieve the files. Our server is on a multi gigabit link and will not limit the speed of your download. Speed will vary depending on the quality of the connection between our server and your client.
Hope this helps.
I am having problems with the FTP server oceans as well. Actually for quite a while. It doesn't always 'connect', but once it connects it is fast (like >700K/s). I have an automated script that downloads and processes the file, but it often fails to download scenes. Running the same command from the command prompt gives the same result. This is using wget on Centos 5.
wget -r --retry-connrefused --tries=20 --timeout=30 --retr-symlinks ftp://oceans.gsfc.nasa.gov:21/MODISA/XM/xxxx/xxxx/A2012142125500.L2_LAC.xxxx.hdf
--2012-05-22 15:40:50-- ftp://oceans.gsfc.nasa.gov/MODISA/XM/xxxx/xxxx/A2012142125500.L2_LAC.xxxx.hdf
Resolving oceans.gsfc.nasa.gov... 220.127.116.11
Connecting to oceans.gsfc.nasa.gov|18.104.22.168|:21... connected.
Logging in as anonymous ...
Error in server response, closing control connection.
And then it retries for another 19 times without success.... It 'hangs' after 'Logging in as anonymous' before it gets the error in server response.
Is there any preferred wget command?
Hmmmmmm, celebrated too early. The ftp server is stuck again...
Wow, that works a lot better! Is it also possible to get a directory listing of my subscription through the oceandata.sci.gsfc.nasa.gov route? Thanks!
It's on the todo list, unfortunately it's a looong list....
We have something in place for extracted data orders, which are staged on our distribution servers (a file called http_manifest.txt would reside in the order subdirectory).
However, nothing is in place for subscriptions...yet.
OK, not a biggie. I am using ncftpls now, instead of wget, and that seems to work a bit more reliable.
I must say, going through http instead of ftp for downloading my scenes make a huge difference. It is much more responsive! I'm flying through my files now!
Thanks for your tips!