Hi there,

all downloads of MODIS L0 files that my servers attempt via http://oceandata.sci.gsfc.nasa.gov/cgi/getfile fail. After about 30MB have been downloaded I get a "connection timed out". I first received this error on Tuesday Sep 14 at about 13:08 UTC.

Any ideas what is causing this?

Cheers, Stefan

Stefan

I am having similar problems.
Download stops after approximately 80 MB.
Assume I run into problems later because I have a higher download speed.

Regards, Gerard

we are having the same problems as well... it seems like a general issue.

Cheers,
Mads

The problem is outside of our project.  We have contacted the network administrators in charge of
out connection to the outside world and asked that they investigate the problem.

I will post a response once we get one.

Regards,
Sean

The firewalls between the OBPG and the outside were rebooted at 5:15pm (EDT)
Thurs, Sep 16, 2010.   At 5:30pm connectivity was restored.                                                               
                                                                                         
There was a problem with packet loss and we could not find a problem                     
on the OBPG side. We talked with the network engineers responsible for
these firewalls and speculated the problem was with either a router or a
firewall they maintained.                               
                                                                                         
Everything should be back to normal now. We will monitor the network                     
to see if any packet loss remains.                             

Regards,
Sean

Hi all.
Has this possibly happened again?
We get incredibly fast download using the http getfile method, but then stalled downloads from OBPG. (and not other site).

Cheers
Matthew Slivkoff

I'm getting failures on MERIS data as well. For instance:
curl --output /optics1/scratch/data/meris/l1b/2011/329/M2011329151028.L1B_FRS.bz2 http://oceandata.sci.gsfc.nasa.gov/cgi/getfile/M2011329151028.L1B_FRS.bz2
hangs up. Has been happening for a couple days now, and on different files.

Thanks,
Brock

We place automatic blocks on sites that use download
accelerators.  (The blocks are released after 15 minutes.)
If you do not think that this is the cause of your download
problems, then let us know which IP addresses you are
coming from, and we can investigate further.

Norman

I have a cron that runs once every 6 hours. It has been running for over a year without issue.
This is the host that does all the d/l ing.

[root@optics ~]# host optics.marine.usf.edu
optics.marine.usf.edu has address 131.247.136.190

Hi Brock,

We checked and you are not getting blocked on our end.
I had a look at our access logs and it appears that you
are only getting about two thirds of the MERIS file that
you used as an example.  It could be that the connection
between you and us is failing somewhere.  You should
be able to resume your downloads in these cases.  Here
are two suggestions from our systems expert.

   If he is using wget he can continue the download with "wget -c
   filename". With curl that would be "curl -C - filename"

Norman

Hi all,

We have had a stable system downloading twice-daily for quite some time too, until approximately 5-6 days ago.

Our code cannot successfully resume using wget.
Sure, wget resumes, but then the bzip file has gibberish in it and won't decompress.
Regardless of our particular circumstance, it seems to me that this resuming is fixing the symptom and not the problem which has crept in.

Earlier in this thread, I noticed Sean contacted someone from outside OBPG (ie between you and us, but still presumably in NASA) and was able to correct similar symptoms.
Is this worth investigating?

If there's anything we can do to help on our end, please let us know.

Sincerely,
Matt

Perhaps related, perhaps not...

Yesterday, I had about 9 L1A_LAC.bz2 files to download.  I set up a file with the names etc and passed it to w.get with the -i option.  after successfully downloading 2 or 3 files it hung.  Killed it, removed the downloaded files from the input to w.get, restarted w.get and it started right up. 

Not using any accelerators (that I know of and with the -i option it should be sequential not parallel.

IP in question is, as near as I can tell, 24.39.41.76

Hi SeaDAS Team,

I can't get any data via http://oceandata.sci.gsfc.nasa.gov/cgi/getfile for about a week now. IP addresses for attempted downloads are 138.80.80.5, 138.80.80.6, 138.80.80.7, 138.80.80.8, 138.80.80.10 and 138.80.80.11.  Update of LUTs and att/eph files also doesn't work. Can you confirm that these addresses are not blocked at your end?

Thanks, Stefan

Hi Stefan,

These addresses had indeed been blocked because of excessive
numbers of erroneous requests.  They should be unblocked now.

Regards,
Norman

Hi Norman,

transfer seems to be working again. Thanks.

Could you please check three more addresses for me:
138.80.92.65
138.80.92.70
138.80.92.80

I don't know what could have caused the excessive numbers of erroneous requests as I haven't changed anything to my systems for months. If the first address is blocked as well than this is very mysterious as it is my desktop which doesn't do any automatic processing. It might be worth checking the whole range 138.80.80.0/20.

Thanks,
Stefan

Stefan,

Those three IP addresses were also blocked for the same reason.
They are unblocked now.

I had a look at our server access logs and do indeed see a lot
of errors from the IP addresses that you mention.  From my brief
look, the problem appears to be that you are attempting to download
level-0 files before they are available in our archive.  Perhaps you
are using your knowledge that MODIS scenes begin at even
5-minute boundaries and then requesting those files before
we have finished ingesting them.  This results in file-not-found
errors which eventually cause your IP to be blocked.

You may want to try one of the approaches suggested here instead.
http://oceancolor.gsfc.nasa.gov/forum/oceancolor/topic_show.pl?tid=3081

Regards,
Norman

Thanks Norman,

yes, I'm predicting which L0 files should be there (and cover my area of interest) and then try to download them. Some of the Aqua files didn't come through last week which obviously caused a larger number of file-not-found errors. I'm trying only twice per day to access a certain file. So the number of errors shouldn't have been too large.

When I have put my download systems together a few years ago there was no file_search option (or at least I wasn't aware of it). So I have chosen the above approach which worked well so far. I will have a look into using file_search to avoid these errors.

Apart from the L0 files I'm also downloading attitude/ephemeris files through the SeaDAS L1b processing scripts. The attitude/ephemeris files become available later than the L0 files. So usually my first attempt of processing a L0 file to L1b fails. Are the SeaDAS L1b scripts using the file_search mechanism as well? Or might they be creating file-not-found errors as well?

Cheers, Stefan

Stefan,

The cut cable from Svalbard that interrupted the MODIS
data flow probably pushed your error rate above our
threshold -- resulting in your IP being blocked.

I will leave your SeaDAS questions to our local SeaDAS
experts.

Norman

Stefan,

Prior to SeaDAS version 6.2 update 2, the processing script that identified the att/ephem files did 'guess', so it would be possible
that it would not find the expected files.  With update 6.2u2, the identification of att/ephem files is done via a database lookup,
so it will only find files that exist.

Sean

Does this all have something to do with not being able to unzip .gz files.  When I run "gunzip *.gz" I am receiving the following error messages:

gzip: A2011152184000.L2_LAC_OC.x.hdf.gz: invalid compressed data--crc error

gzip: A2011152184000.L2_LAC_OC.x.hdf.gz: invalid compressed data--length error

I've tried multiple datasets/files!

any help would be great!

Thanks

Erin

No.  Either the files were corrupted on download, or your FTP client retrieved the file in ASCII mode
when it needs to be in BINARY mode (I'm assuming from the file name that these were extracts staged
on our FTP server...)

Sean