Not logged inOcean Color Forum

The forum is locked.

The Ocean Color Forum has transitioned over to the Earthdata Forum (https://forum.earthdata.nasa.gov/). The information existing below will be retained for historical reference. Please sign into the Earthdata Forum for active user support.

Up Topic Products and Algorithms / Satellite Data Access / Problems with downnloads (locked)
- By lucasbarbedo Date 2014-02-27 09:11
Problems with downnloads
I and my colleagues in the same institute are experiencing problems when downloading data from ftp oceancolor.

When trying to download data from MERIS N1, MODIS L1A, the seadas 7.0.2 program and process data MODIS L1A to L1B by seadas have had problems, the connection fails or takes too long sometimes not complete downloading data, I also believe that the process for L1B is compromised due to downloading auxiliary data.
Is there any impediment in transferring by FTP?
- By seanbailey Date 2014-02-27 09:45
None that we are aware.

Please send an email to ftpobpg1 at seawifs.gsfc.nasa.gov with the following information:

1) IP address(es) from which access to our servers was attempted
2) date and time of the attempt
3) URLs attempted

We will examine our logs to see if there are any issues related given the provided information
- By gnwiii Date 2014-02-27 10:04
We have much the same set of problems at my institute in Canada.  

Unfortunately, the internet is not what it used to be, and "institute" sites everywhere are a target for "science haters".  Ftp was never intended to handle today's internet:

  The specification for the File Transfer Protocol (FTP) contains a
   number of mechanisms that can be used to compromise network security.
   The FTP specification allows a client to instruct a server to
   transfer files to a third machine.  This third-party mechanism, known
   as proxy FTP, causes a well known security problem. 
   -- <FTP Security Considerations (May 1999)>

My impression is that the file transfer problems come from the difficulties of two network fortresses that are constantly fending off attacks by adjusting the firewall shields while trying to preserve some ability to move legitimate data.

Maybe we need a "FedEx net" where disk arrays are shipped from institute to institute.

   Never underestimate the bandwidth of a station wagon full of tapes hurtling
   down the highway. –Andrew Tanenbaum, 1981
- By lucasbarbedo Date 2014-02-27 14:58 Edited 2014-02-27 15:02
A few days ago that we are trying to process the data from the
MODIS L1B to L1A level, in addition to downloading the files.

I sent the data downloads as suggested.

It seems that the downloads normalized and I can now download the
MODIS L1A and seadas 7.0.2. Wonder!

However, finally there's something wrong with the L1A to L1B
processing, is it caused by this problem seems on the internet?
Usually below logs processing failed.

more LogReport.T2010119165500.L1B

****************************************
BEGIN_PGE: Thu Feb 27 16:10:49 2014
MSG_TAG: 11
FILE: LogReport.T2010119165500.L1B
LOGGING: status message logging enabled
TRACE_LEVEL: tracing disabled
PID_LOGGING: disabled
DISABLED_LEVELS: none
DISABLED_SEEDS: none
DISABLED_CODES: none
THREAD-SAFE MODE:  disabled
TOOLKIT_VERSION: SCF  TK5.2.18
****************************************

Thu Feb 27 16:10:49 2014

Read_LUT_Tables():MODIS_F_FILE_NOT_OPENED:295735308
Could not open file for SD read access.
Call to SDstart() failed.
File LUN:  700050
File type: Reflective_Lookup_Tables_file
The file may be missing, corrupted or not an HDF-4 file.
==========================================================

Thu Feb 27 16:10:49 2014

File not opened running...MOD_PR02. TIME:Thu Feb 27 16:10:49 2014
  Operator Actions:
  Check previous messages to identify LUN of the file and other
possible causes.
  Check the PCF file to ensure that file names and directories are correct.
  Contact MCST.

more LogStatus.T2010119165500.L1B

****************************************
BEGIN_PGE: Thu Feb 27 16:10:49 2014
MSG_TAG: 11
FILE:  LogUser.T2010119165500.L1B_LAC
ING: status message logging enabled
TRACE_LEVEL: tracing disabled
PID_LOGGING: disabled
DISABLED_LEVELS: none
DISABLED_SEEDS: none
DISABLED_CODES: none
THREAD-SAFE MODE:  disabled
TOOLKIT_VERSION: SCF  TK5.2.18
****************************************

Read_LUT_Tables():MODIS_F_FILE_NOT_OPENED:295735308
Could not open file for SD read access.
Call to SDstart() failed.
File LUN:  700050
File type: Reflective_Lookup_Tables_file
The file may be missing, corrupted or not an HDF-4 file.

:MODIS_F_FILE_NOT_OPENED:295735308
File not opened running...MOD_PR02. TIME:Thu Feb 27 16:10:49 2014
  Operator Actions:
  Check previous messages to identify LUN of the file and other
possible causes.
  Check the PCF file to ensure that file names and directories are correct.
  Contact MCST.

more LogUser.T2010119165500.L1B_LAC

****************************************
BEGIN_PGE: Thu Feb 27 16:04:35 2014
MSG_TAG: 11
FILE: LogUser.T2010119165500.L1B_LAC
LOGGING: status message logging enabled
TRACE_LEVEL: tracing disabled
PID_LOGGING: disabled
DISABLED_LEVELS: none
DISABLED_SEEDS: none
DISABLED_CODES: none
THREAD-SAFE MODE:  disabled
TOOLKIT_VERSION: SCF  TK5.2.18
****************************************
- By gnwiii Date 2014-02-28 07:45
I had this problem a couple days ago.   IN the PCF file you can find the name of the "Reflective_Lookup_Tables_file".  In my case, this file could be shown to be defective using the ncdump_hdf command:


$ ncdump_hdf -h $OCSSWROOT/run/var/modisa/cal/OPER/MYD02_Reflective_LUTs.V6.1.17.21_OC.hdf
    *** ncdump_hdf: ncopen failed on /Volumes/Data-Ambrosia/SeaDAS/OSX/seadas-7.0.2/ocssw/run/var/modisa/cal/OPER/MYD02_Reflective_LUTs.V6.1.17.21_OC.hdf


I found proper HDF file with the same name on another system, so just copied it over and finished the calculations.  I did run "update_luts.py" on the problem system, so it looks like "update_luts.py" could be more helpful in the case of corrupt files.   If you don't have a good version of the file, just delete it and run "update_luts.py" again.
- By seanbailey Date 2014-03-02 14:30

> so it looks like "update_luts.py" could be more helpful in the case of corrupt files.


Indeed, there is no check for corrupted files...we'll investigate the possibilities. 

The download function used by update_luts.py does it's best  to retrieve the file,
retrying failed attempts and continuing incomplete downloads, but it has no logic
to test the downloaded file.  There are limits to how hard it will try, so perhaps it
hit the limit but had not fully retrieved the file.  We may be able to detect that
situation and remove the partial download (and alert the user :smile:)

Regards,
Sean
- By gnwiii Date 2014-03-02 14:49
My first idea when discovered that ncdump_hdf -h couldn't read the file was to run update_luts.py again.  I should have deleted the bad file before running  update_luts.py.  It could be helpful if the file was downloaded under a different name, e.g. .hdf.dnld and only renamed if the download seemed to be successful. 

It is certainly not helpful to fail silently when a download is known to have failed (but also not helpful when work on science code competes with work on ways to get around internet breakages).  There really should be robust, public, and widely used, python code to fetch a file reliably or tell what the problem is when a download fails.
- By lucasbarbedo Date 2014-03-06 09:09
How Should i proceed to finish the MODIS L1a to L2 process on seadas?

If I understand I need clear and remove the auxiliary data and reprocess the MODIS images.
Can I applicate this commands?
rm softwares_sens_remoto/seadas-7.0/ocssw/run/data/modist/aerosol/*hdf
rm softwares_sens_remoto/seadas-7.0/ocssw/run/data/modist/cal/*hdf
rm softwares_sens_remoto/seadas-7.0/ocssw/run/data/modist/ocr_vc/*hdf
rm softwares_sens_remoto/seadas-7.0/ocssw/run/data/modist/rayleigh/*hdf
rm softwares_sens_remoto/seadas-7.0/ocssw/run/data/modist/*parrm softwares_sens_remoto/seadas-7.0/ocssw/run/data/modist/*dat
- By seanbailey Date 2014-03-06 09:47
No, removing those would be bad and prevent all processing.
All you need to do is remove the apparently buggered L1B LUT files.
These reside in the <ocsswroot>/run/var/modis[a|t]/cal/OPER directory.
Once removed, rerun update_luts.py terra and  update_luts.py aqua

Sean
- By lucasbarbedo Date 2014-03-13 14:22
I try to ping on the ip address 169.154.128.84 and I get as statistics
.....
64 bytes from 169.154.128.84: icmp_seq=27 ttl=52 time=198 ms
64 bytes from 169.154.128.84: icmp_seq=29 ttl=52 time=198 ms
64 bytes from 169.154.128.84: icmp_seq=30 ttl=52 time=199 ms
64 bytes from 169.154.128.84: icmp_seq=31 ttl=52 time=200 ms
64 bytes from 169.154.128.84: icmp_seq=32 ttl=52 time=199 ms
^C
--- 169.154.128.84 ping statistics ---
33 packets transmitted, 23 received, 30% packet loss, time 32081ms

If I use wget to download L3 data from seawifs sensor like
wget http://oceandata.sci.gsfc.nasa.gov/cgi/getfile/S20090322009059.L3m_MO_CHL_chlor_a_9km.bz2 (only for testing a download)

the download starts
.....
Saving to: ‘S20090322009059.L3m_MO_CHL_chlor_a_9km.bz2.2

12% [===>                                   ] 1.388.520   46,2KB/s   in 12s

however, in a few seconds I get this message

2014-03-13 15:12:56 (109 KB/s) - Read error at byte 1388520/11540113 (Connection reset by peer). Retrying.

--2014-03-13 15:12:57--  (try: 2)  http://oceandata.sci.gsfc.nasa.gov/cgi/getfile/S20090322009059.L3m_MO_CHL_chlor_a_9km.bz2
Connecting to oceandata.sci.gsfc.nasa.gov (oceandata.sci.gsfc.nasa.gov)|169.154.128.84|:80... connected.

and so on...

In general I can't conclude the download....
- By seanbailey Date 2014-03-13 15:15
wget has a continuation option (-c). 
I've posted an example script that uses wget to grab a list of files from our servers - with error checking and using the continuation option.
You might want to give it a try.  More information on it is in the FAQ I wrote to go with the script.

Sean
- By lucasbarbedo Date 2014-03-17 13:41
Sean,
The big problem is that every time we make downloads here INPE.
We have problems and fall downloads, download files from the L3 oceancolor site, or when we need to ancillary data processing Seadas.

We think that when we reach a limit of downloaded data rate at the same instant, the IP entered a kind of list of spawn, as if we were rakers.
- By seanbailey Date 2014-03-17 16:41
There are limits imposed on data access, but these are reasonable limits and are only designed to prevent DDOS attacks on our systems.
If you have multiple systems behind a single firewall, and all attempt to connect to our systems at the same time, this can exceed the
imposed limits.  If this is not the case, and you are accessing from a single system, if you have any "download accelerators" in use, or
spawn a lot of simultaneous processes, this too can exceed the limits.  If neither of these are the case, send a email to ftpobpg1 at seawifs.gsfc.nasa.gov
with a description of the problem (include details such as time of access, URLs attempted, and your external IP address).

Regards,
Sean
- By towens Date 2014-03-17 16:57
The route from INPE appears to use the RedCLARA link which is reporting
problems today. Try a traceroute to 169.154.128.58 and see if any of the
hops report saopaulo-miami.core.redclara.net

see below:
The Internet2 NOC has reported another issue with the RedCLARA link.

Outage - I2 IP Peer RedCLARA (LOSA)
http://noc.net.internet2.edu/

Tommy
Up Topic Products and Algorithms / Satellite Data Access / Problems with downnloads (locked)

Powered by mwForum 2.29.7 © 1999-2015 Markus Wichitill