Not logged inOcean Color Forum

The forum is locked.

The Ocean Color Forum has transitioned to the Earthdata Forum (https://forum.earthdata.nasa.gov/)

Up Forum
Search
Topic FTP Problems? By mcsorley Date 2008-05-22 18:16
Why can't I connect from my linux machine?

It has been brought to our attention that some linux kernels after 2.6.7 have had window scaling turned on by default. Some routers on the net are rewriting the window scale TCP option on SYN packets as they pass through. The result is a misunderstanding over the real size of the receive window between our system and yours. If this is happening with your connection then you will not be able to contact our ftp servers at all.

Anybody running a current kernel who is having trouble connecting can try this work around to see if it helps. The following command will need to be executed with elevated privileges (root). This will turn off window scaling for that machine.

echo 0 > /proc/sys/net/ipv4/tcp_default_win_scale

After running this command try the ftp connection again. If it works then that was the problem. Add the following line to your / etc/sysctl.conf file so it executes on reboot.

net.ipv4.tcp_default_win_scale = 0
Topic FTP Access By mcsorley Date 2007-12-27 15:04
We have not made any changes lately that would effect ftp transfers. If you can get in with an FTP client on any machine that at least tells us that your not being blocked by any filters. Just make sure your using passive FTP to connect to our system and if you want to try to troubleshoot have a look at our FTP FAQ.

http://oceancolor.gsfc.nasa.gov/forum/oceancolor/topic_show.pl?pid=4395;hl=ftp#pid4395
Topic "Alarm Clock" kills ftp session? By mcsorley Date 2007-01-23 12:05
Barry,

I noticed you said when you issue the passive command it turns passive mode off. You want to have passive mode on. When you start your session it looks like passive mode is the default setting. So don't type passive at the start of the ftp session. Or if you do just type "passive" again and make sure it says "passive mode on" or something like that.

...joe
Topic "Alarm Clock" kills ftp session? By mcsorley Date 2007-01-23 10:07
Barry,

We don't have anything built into our ftp servers that will kill your download on a set time like an "Alarm Clock". Feel free to post the output of your full ftp session with the error and we will see if we notice any thing funny. The more information we have about the situation the better. Things like: How long can you go until the message pops up? The ftp client name? When did it start happening? 
Topic MODIS Quicklook products not available By mcsorley Date 2006-10-21 17:04
The quicklook data has been fixed. Thanks for being patient.
Topic MODIS Quicklook products not available By mcsorley Date 2006-10-20 16:19
Jacques,

I don't beleive it will be up before the weekend. We are looking into it and will try to get it resolved as soon as possible.

...Joe
Topic unable to login to ftp oceans.gsfc.nasa.gov By mcsorley Date 2006-06-23 10:06
Brock,

Check some of the timeout settings wget offers. If you set your own timeout limits you will never have to worry about the other sides limits. Some of wget's timeout settings worth checking out are: --timeout, --connect-timeout, --read-timeout, --wait, and --waitretry. Descriptions of all of these options are in the wget man page (man wget). If you do this things will begin to fail gracefully and not get stuck.

...Joe
Topic unable to login to ftp oceans.gsfc.nasa.gov By mcsorley Date 2006-06-22 16:16
Brock,

Seashell is at the login limit right now. If your script tries to connect over and over again while seashell is logged in it will hang because of the limits we have set on the servers. fox should be ok to login while seashell is logged in. Check the ftp man page (man ftp) on specificying a username an password. You might also try "ftp -n oceans.gsfc.nasa.gov".  -n    Restrains ftp from attempting ``auto-login'' upon initial connection. Only try the -n if your client supports it. The man page will tell you about this.
Topic Do you limit bandwidth in any way? By mcsorley Date 2006-06-21 13:00
Do you limit bandwidth in any way?

There are currently no limits or caps on bandwidth to our site. The only limit is the slowest network between you and us.

Please see the FAQ question "Why are my downloads going so slow?" for help on slow connections.
Topic Why are my downloads going so slow? By mcsorley Date 2006-06-21 12:49
Why are my downloads going so slow?

It is possible the network your connecting from could be limiting your bandwidth or could be very busy at the time you are trying
to connect. There are many reasons why your downloads could be going slower than they usually do. When connecting to our
servers your data will have to travel through many networks to get here. If any network in the chain is having problems then your
connection speeds will be affected. Users outside of the United States are affected by this more than anyone.

There are websites that let you check general network latency at major points around the country. You can check here to
see there are any big problems going on that may be affecting your transfer.

http://www.internettrafficreport.com/

http://www.internetpulse.net/

For these charts to be helpful you will need to know who your Internet Service Provider is and which network provider they use for
their connection to the internet. Your systems administrators can usually provide you with this information. 
Why are my connections hanging when I try to retrieve a file?

You are trying to connect beyond the host connection limit set by OBPG. Any
subsequent connection after the limit will not be answered by the servers. FTP
client programs see this as a connection that never closes. So they will hang
on those connections. The best way to have this not happen is to only connect
to the FTP server a couple of times. Constant requests to the server after the
host limit has been reached will fail. To have this fail gracefully set the
connection timeouts in your FTP client programs to fail after a short period of time.

Your connections could also be hanging when trying to connect to the webservers to
download data. The webserver will limit your ability to download data by time
between requests. It also has the host connection limits described above.
Are their limits or restrictions put in place for data retrieval?

Yes there are limits and restrictions in place for data retrieval. The Ocean
Color community is compromised of thousands of people from all around the world.
To give everyone in the community equal access to Ocean Color data there needs to be
limits set so one person does not prevent another from retrieving his or her
data.

The primary focus of the restrictions are based upon amounts of connections and
time between connections.

Examples of ways to get blocked from data retrieval.

1. Retrieving the same file over and over again in a loop.
2. Flooding our network with connection attempts.
3. Using download accelerators.
4. Out of control scripts.

Where people get in trouble with the restrictions are when they write very
aggressive scripts. Scripting your data retrieval is a great idea. But you need
to bear in mind that when your script does the file retrieval too quickly
you will begin hitting the retrieval limits set by OBPG. Try to keep it to a
couple of connections at a time.

For example let's say you have a script that logs in and retrieves a FTP
directory listing of 1000 files. Then it executes an FTP request for each file
it just got a listing for. This in turn would open 1000 connections to the FTP
or web server at one time. Doing this in a loop until you successfully retrieve
all the files puts a heavy load on our servers. You are limited by the amount
of times you can login to the FTP server. All of the extra connections will just
be denied. The constant barrage of hundreds of connections at a time from
one host can also be viewed as a denial of service attack.

If you're going to try to script your data retrieval here are some suggestions
that will help insure the successful retrieval of that data.

1. Use scriptable friendly programs like curl, wget or ncftpget. They have many
settings like timeout periods you can adjust to help things fail gracefully.

2. Use FTP programs that support resume functions. This will help you finish
downloading any file if your connection gets cutoff.

3. Try to keep your FTP connections to a couple at a time. Trying to login as
many times as possible over and over again will get you blocked at some point. A
couple of connections at a time will ensure you can login to get your data as
long as the server is not full.

4. Write your scripts with connection limits in mind. Pounding away with
hundreds of connections at once is useless. Put in sleep states between
file retrieval attempts. Keep track of the files you need and the files you
currently have. Request the files from your list a couple at a time. Then
after one of those files are done downloading request the next one.
What do I do when I get a decompression error on a data file?

First make sure your FTP transfer is done in binary mode. Most FTP clients
are pretty good about picking the right mode but just to be sure before you
download the file type the word "binary" at the FTP prompt. In your GUI FTP
clients make sure your area with transfer type is set to "binary". You can tell
if it worked by scrolling back through the transfer window and looking for the
word "binary".

Second thing to check is correct file size. Make sure the file transfered
completely. Login to the FTP servers and issue an "ls" command. This will
provide you with a file size in bytes of all of the files. Check your local
file size with the one on the FTP server. If they do not match then your file
transfer did not complete.

Our FTP servers support the resume function. This means if your FTP client
supports this you can resume downloading a file where you left off if your
connection gets cut part of the way through. This also means you can have
partially downloaded files that look like they could be nearly the right size.
The only way to tell if the size if correct is a file size comparison of
server file size to local file size.
What are the ways can I retrieve the satellite data?

There are a few ways of retrieving data from OBPG. You can use the browseable
web interfaces or you can use FTP.

If you would like to use the the web interfaces there are links on the main page
http://oceancolor.gsfc.nasa.gov/ to the level 1/2, and 3 browsers.

If you would like to retrieve your data via the File Transfer Protocol (FTP)
please refer to the link http://oceancolor.gsfc.nasa.gov/ftp.html. This page has
all of the links that will provide you with direct ftp access to data products
organized by satellite.
Topic FTP Problems? By mcsorley Date 2006-06-01 11:13
What do I do if my FTP program does not support passive FTP?

If you are sure your FTP program does not support passive or if you can't figure
out if it does then don't fret. There are lots of free FTP programs that support
passive mode. Below is a list of FTP programs (clients) by operating system that
are known to support passive mode.

----------------------------------
Command Line Clients
---------------------------------
Linux -                                           ncftp, ftp included in Linux NetKit version .17 and above, wget, and yafc.
IRIX 6.0 - 6.5.8 -                            ncftp.
IRIX 6.5.14 and up -                       ncftp or ftp included in IRIX 6.5.14.
SUN Solaris 7 and 8 -                     ncftp.
SUN Solaris 9 -                              ncftp or ftp included in SUN OS 9.
FreeBSD 2.2.8 and up -                  ncftp or ftp included in FreeBSD.
OpenBSD 2.7 and up -                    ncftp or ftp included in OpenBSD.
Windows 95,98,2000,ME,NT & XP - ncftp or Kermit 95 2.0.
Mac OS X 10.2 and up-                   ncftp or ftp included in OS X.

-----------------
GUI Clients
-----------------
Linux or FreeBSD -      gFTP or konqueror.
Windows -                  FTP Explorer, Filezilla, coreftp.
Mac OS X -                Cyberduck.

--------------------------------------------------------------------------
The following are known not to support passive ftp.
--------------------------------------------------------------------------
- FTP program that comes with SUN Solaris 7 and 8.
- Internet Explorer's 5.1 and above default settings. Needs to be turned on in
  IE settings.
- Internet Explorer 5 set to passive mode and "file explorer" mode.
- FTP program that comes with IRIX 6.5.14 and below.
- FTP command line program that comes with windows 95, 98, NT, 2000, and XP.

OBPG suggests using ncftp as your FTP client. It is command line based but can
be used on almost all operating systems. It is very friendly to scripting
(ncftpget, ncftpls, ncftpput, ncftpbatch), free, and has automatic fallbacks to passive
if active does not work. It can be downloaded from http://www.ncftp.com/.

The second reason this could be failing is that your local network firewall/proxy
does not permit passive ftp connections. You will need to ask your local network
administrator if this is the case.

Always check with your local systems administrators first if you're having any problems
connecting to remote sites. They know the network configuration and restrictions better
than anyone.
Topic FTP Problems? By mcsorley Date 2006-06-01 11:03
Why is it when I login to you're FTP servers and try to get a file it fails?

There are a few reasons why this could be happening. First and foremost you
must make sure your FTP client (program) is using passive mode to connect to our
servers. FTP has 2 primary modes it can use. Active and passive. Active is used
by many FTP clients as the default transfer method. The problem with this is
active FTP does not work well with firewalls. When you login successfully
but your connection hangs on a "get" or "ls" command you're probably using FTP in
active mode. If this is the case then the message that will be returned to your
FTP client will be something to the effect of "501 EPRT: Operation not permitted"
or "501 PORT: Operation not permitted".

Today most modern FTP clients can be used in either active or passive FTP modes.
You just have to tell the client which mode to use. Most window based clients
have a configuration settings section where the mode can be set. Try searching
through the help section of the program with the word "passive" or "firewall" if
you have trouble finding the section with the setting. The command line FTP
clients use command line switches to set passive mode. If you're using a Unix type
operating system use the systems man pages (Ex. man ftp) to check the switches
and options of the FTP client you're trying to use. If you don't see anything in
these sections with the words passive or firewall then it's a good possibility
that FTP client does not support passive.

Please refer to this website for a good explanation of active vs passive FTP.

http://slacksite.com/other/ftp.html
Topic FTP Problems? By mcsorley Date 2006-06-01 10:50
Why is my FTP connection not working?

We need to find out what part of the process is failing.

1. Is your network connection working?

Check that your network connection works by hitting a few websites you've never
been to before. If they come up then try an anonymous login to a few public FTP
servers (ftp.openbsd.org, ftp.microsoft.com). After you login make sure you can
do an directory listing with the "ls" command. If this works move on to step 2.

2. Can you connect to the Ocean Color servers at all?

There are a few possibilities on why you can't contact the Ocean Color servers.

The first possibility is that there is network connection problem to the main
distribution site.

Try to pull up the main Ocean Color website. Located at
http://oceancolor.gsfc.nasa.gov. If you can access that site then the main
distribution network is up and running correctly.

The second possibility is that the server you're trying to contact is temporarily
down.

If you try you're FTP login and get nothing back it could be that the FTP server
is down temporarily. To see if there are problems with any system please
subscribe to the OBPG data processing/distribution list.

This can be done from the link:
http://oceancolor.gsfc.nasa.gov/cgi/ocmailsubscribe.cgi?list=obpg-notify.

You could also check the color of the stop light on the main Ocean Color
website. Anything other than green means there is a problem with the
distribution system. If the stoplight area mentions problems with an FTP
server please wait for the light go back to green before attempting another
FTP connection. If there is no description of the problem but the light is yellow
please wait for the light to return to green. Then try your connection again.

The third possibility is your public ip address has been placed on a block list.

Our servers are on a block of ip addresses assigned by NASA. Therefore we are
subject to all filters NASA has put in place. NASA has certain ip addresses it
is blocking for a variety of reasons. It is possible you are connecting from one
of those addresses.

It is also possible you have been put on OBPG's personal block list. This
results from our detection of activity we believe had malicious intent. The
usual cause of this is a script that is trying to retrieve files too quickly.

It is unusual for someone to be put on a block list. So contacting us about it
should be a last resort effort.

Before we check our block lists make sure the connection problem is not due to
something other than a block list on our side. Check with your systems administrators
to see if they have blocked you're access to our site either by ip address or by name.

If you would like us to check if your ip address has been put on a block list
please e-mail ftpobpg1 at seawifs.gsfc.nasa.gov with the public ip address you're
connecting from. To find out what your public ip address is please go to the
website  http://www.ipchicken.com. Then send us the number that appears under
the title "Current IP Address".

If you can connect to the Ocean Color web server or can connect to the FTP
server move onto step 3.

3. Can you login to the FTP server?

For the first part of FTP (control connection) to work correctly you need to be able
to connect to the FTP server. After connecting you should be presented with a
banner message and login prompt.

From a command line client it is easy to tell if the FTP login is working. You
will be presented with a NASA login banner. Then a username and password prompt.
You login anonymously with the username "anonymous" or "ftp". The password is
your e-mail address. After that you should be at an FTP prompt. If you're using a
GUI client set it to login anonymously. Then to see if you logged in correctly
there is usually a window that scrolls all of the commands issued by the client.
You will have to scroll back  through this window to see if the login completed
correctly.

If you can login successfully and get to a FTP prompt move onto step 4.

4. Can you get a file listing or retrieve a file?

After logging in to the FTP server and getting to a prompt you should be able to
issue a "ls" command from a command line client. GUI clients issue this command
for you automatically when you login successfully. If this command is issued and
you get nothing back you might not be using the  correct mode of FTP transfer.
To check this please refer to the question "Why is it when I login to your FTP
servers and try to get a file it fails?" in the post below.

After you have read this and verified you're using the correct client and FTP
settings please move to step 5.

5. All other steps have checked out and are working. Now what?

You should be to the point where you've made sure your FTP client supports
passive FTP, that it is turned on, have successfully logged in, and have tried
to retrieve a file or get a directory listing. At this point if retrieving a
file or getting a directory listing fails then there is not much you can do from
here.

Our final suggestion is that you get in contact with your systems and network
administrators. They deal with these types of problems on a daily basis. They usually
have better access to information about the network setup at your location and
can provide you with more insight into why FTP may not be working correctly.

If your systems administrators say everything should work correctly and can reproduce
the same problem you're having then your last resort is sending OBPG an e-mail.

If you are going to send us an e-mail about an FTP problem we ask that you take
the time to read the whole FAQ first. In the body of the email you will need to
answer the following questions. If you do not answer each and every
question/request we will not be able to help you with your problem.

- What is the public IP address you're coming from? If you don't know please visit
  http://www.ipchicken.com. Copy and paste into the e-mail the big number under
  the words "Current IP Address".

- What is the data you are trying to retrieve?

- Where is the data you are trying to retrieve? The hostname of the server you
  are trying to contact and (if you have it) the path to the data.
  Example:  ftp://oceans.gsfc.nasa.gov/MODIST/L3BIN/2006/006

- How many times have you tried to retrieve the data?

- What are the exact times you tried to retrieve the data?

- What is the name and version of FTP client (program) you are using?

- What is the name and version of operating system you're using?

- Copy and paste the full text of you're failing FTP session to you're e-mail.

- What is the e-mail address of you're systems administrator?

Send the answers to these questions and a description of the problem to: ftpobpg1 at seawifs.gsfc.nasa.gov

Powered by mwForum 2.29.7 © 1999-2015 Markus Wichitill