Dear,
According to the seadas annoucements, a update to the SeaDAS scripts that connect to these websites is necessary(https://oceancolor.gsfc.nasa.gov/forum/oceancolor/topic_show.pl?tid=6424), but my update operation is failed both GUI and command line.
In GUI, a execution log is:
[/tmp/install_ocssw.py, --install-dir=/home/rslakes/optics/seadas-7.3.2/ocssw, --goci, --meris, --aqua, --terra, --oli, --viirsn, --git-branch=v7.3]
/tmp/install_ocssw.py: 1: /tmp/install_ocssw.py: cannot open html: No such file
/tmp/install_ocssw.py: 2: /tmp/install_ocssw.py: Syntax error: redirection unexpected

In command line, the update throw an exception until step 9, details are:
Installing ocrvc (5 of 9)
Installing bin (6 of 9)
fatal: unable to access 'http://oceandata.sci.gsfc.nasa.gov/ocssw/bin-linux_64.git/': Recv failure: Connection reset by peer
Error - Could not run "cd /home/rslakes/optics/seadas-7.3.2/ocssw/run/bin/linux_64; git pull --progress -q > /dev/null"

I can't find the related topics in forum after research, so could you give me some suggestions?

Best Regards,
zhigang

I had a somewhat similar problem (slightly different error message). I am using Mac OS 10.12.1 (Sierra).

[sylviaearle : ~] monger 3.2$ $OCSSWROOT/run/scripts/install_ocssw.py
Installing bundles.sha256sum (1 of 9)
URL transformed to HTTPS due to an HSTS policy
--2016-12-23 08:34:52--  https://oceandata.sci.gsfc.nasa.gov/ocssw/bundles.sha256sum
Resolving oceandata.sci.gsfc.nasa.gov (oceandata.sci.gsfc.nasa.gov)... 169.154.128.84, 2001:4d0:2418:128::84
Connecting to oceandata.sci.gsfc.nasa.gov (oceandata.sci.gsfc.nasa.gov)|169.154.128.84|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 2578 (2.5K) [text/plain]
Saving to: ‘bundles.sha256sum’

bundles.sha256sum                             100%[==============================================================================================>]   2.52K  --.-KB/s    in 0s     

2016-12-23 08:34:52 (205 MB/s) - ‘bundles.sha256sum’ saved [2578/2578]

Installing common (2 of 9)
Installing OCSSW_bash.env (3 of 9)
URL transformed to HTTPS due to an HSTS policy
--2016-12-23 08:34:53--  https://oceandata.sci.gsfc.nasa.gov/ocssw/OCSSW_bash.env
Resolving oceandata.sci.gsfc.nasa.gov (oceandata.sci.gsfc.nasa.gov)... 169.154.128.84, 2001:4d0:2418:128::84
Connecting to oceandata.sci.gsfc.nasa.gov (oceandata.sci.gsfc.nasa.gov)|169.154.128.84|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 2894 (2.8K) [text/plain]
Saving to: ‘OCSSW_bash.env’

OCSSW_bash.env                                100%[==============================================================================================>]   2.83K  --.-KB/s    in 0s     

2016-12-23 08:34:53 (276 MB/s) - ‘OCSSW_bash.env’ saved [2894/2894]

Installing README (4 of 9)
URL transformed to HTTPS due to an HSTS policy
--2016-12-23 08:34:53--  https://oceandata.sci.gsfc.nasa.gov/ocssw/README
Resolving oceandata.sci.gsfc.nasa.gov (oceandata.sci.gsfc.nasa.gov)... 169.154.128.84, 2001:4d0:2418:128::84
Connecting to oceandata.sci.gsfc.nasa.gov (oceandata.sci.gsfc.nasa.gov)|169.154.128.84|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 3358 (3.3K) [text/plain]
Saving to: ‘README’

README                                        100%[==============================================================================================>]   3.28K  --.-KB/s    in 0s     

2016-12-23 08:34:53 (291 MB/s) - ‘README’ saved [3358/3358]

Installing ocrvc (5 of 9)
Installing bin (6 of 9)
fatal: unable to access 'http://oceandata.sci.gsfc.nasa.gov/ocssw/bin-macosx_intel.git/': Empty reply from server
Error - Could not run "cd /Applications/seadas-7.3.2/ocssw/run/bin/macosx_intel; git pull --progress -q > /dev/null"
[sylviaearle : ~] monger 3.2$

Any suggestions are appreciated...
Bruce

OK,

Two problems....

1) I failed to push the updated install_ocssw.py script to the distribution server....oops!

2) The installer script needed another modification to switch the git remote repository to the https location.

...well, make that three problems...

3) The GUI is tries to pull from the http location and does not honor the redirect, so the update needs to be done manually
until we get an update out for the GUI.

So, you will need to manually pull down the install_ocssw.py script follow the
manual install/update process (scroll down the page about halfway)

Regards,
Sean

Thanks Sean!  It now works perfectly!   You defiantly gave me a nice Christmas Gift. ;).   I am heading to Fiji on January 4 to teach my satellite training course and I was getting a bit nervous about getting all of this ssl stuff to work nicely.   So I am set now...

Merry Christmas
Bruce

The online version check in the SeaDAS GUI is failing today.  

People encountering problems with git using http URLS can run: git config --global url."https://oceandata.sci.gsfc.nasa.gov".insteadOf http://oceandata.sci.gsfc.nasa.gov.   I think git uses the curl library, so some older distros may encounter problems with curl library versions.

Thanks a lot Sean. The update script works well now.
Thanks for your patience and help all the time.
Merry Christmas!

Best Regards,
zhigang

Confirmed that I am able to do this:

./install_ocssw.py --install-dir=$OCSSWROOT --git-branch=v7.3 --oli

but the Update OCCSW Processors tab returns:

execution exception: java.io.IOException: install_ocssw.py failed with exit code 2.
Check log for more details.
/tmp/install_ocssw.py: line 1: html: No such file or directory
/tmp/install_ocssw.py: line 2: syntax error near unexpected token `<'
/tmp/install_ocssw.py: line 2: `<head><title>301 Moved Permanently</title></head>
'
And:

The GUI needs an update to use https URL's.  Watch for an announcement.

Hi --  My processing script was working last week (Dec 23 around noon eastern time) after Sean updated the install script to handle https, but now things seem to be hanging up when I try to download ancillary data for processing from level 1 to level 2.  I don't recall changing anything  on my end (but I certainly could have overlooked something in my frenzy to get things working) so I was if something on your end might have changed between Dec 23 and now.  Below is the error response when I used modis_GEO.py.  Note my script makes three passes with using modis_GEO.py because of problems with the database file that I encountered in the past.  The first pass is to call:  modis_GEO.py -v -o  fname_geo  fname_l1a.  If I fail to generates a geo file, then second attempt uses: modis_GEO.py -v --refreshDB -o  fname_geo  fname_l1a   and if this fails, I remove the database file $OCSSWROOT/run/var/ancillary_data.db and then call  modis_GEO.py -v -o  fname_geo  fname_l1a.   Below are the error messages for each of these three attempts.  Also, I have recursively opened up write permissions to the ancillary directories.

Thanks for any help/comments
Bruce

>=====>  generating geolocation with modis_GEO.py -v -o  fname_geo  fname_l1a...

Determining required attitude and ephemeris files...
Searching database: /Applications/seadas-7.3/ocssw/run/var/ancillary_data.db

Input file: /rsclass/fiji_test/aqua_l1a_gom/A2016240174000.L1A_LAC.x.hdf
Start time: 2016240174000
End time: 2016240174459

Downloading 'PM1EPHND.P2016240.1200.003' to /Applications/seadas-7.3/ocssw/run/var/anc/2016/240
*** ERROR: The HTTP transfer failed with status code 404.
*** Please check your network connection and for the existence of the remote file:
*** oceandata.sci.gsfc.nasa.gov/cgi/getfile/PM1EPHND.P2016240.1200.003
***
*** Also check to make sure you have write permissions under the directory:
*** /Applications/seadas-7.3/ocssw/run/var/anc/2016/240

Cannot create geolocation from /rsclass/fiji_test/aqua_l1a_gom/A2016240174000.L1A_LAC.x.hdf; exiting.

>=====> attempting modis_GEO.py with --refreshDB ...

Determining required attitude and ephemeris files...
Searching database: /Applications/seadas-7.3/ocssw/run/var/ancillary_data.db

Input file: /rsclass/fiji_test/aqua_l1a_gom/A2016240174000.L1A_LAC.x.hdf
Start time: 2016240174000
End time: 2016240174459

Downloading 'PM1EPHND.P2016240.1200.003' to /Applications/seadas-7.3/ocssw/run/var/anc/2016/240
Connection error, retrying up to 5 more time(s)
*** ERROR: The HTTP transfer failed with status code 404.
*** Please check your network connection and for the existence of the remote file:
*** oceandata.sci.gsfc.nasa.gov/cgi/getfile/PM1EPHND.P2016240.1200.003
***
*** Also check to make sure you have write permissions under the directory:
*** /Applications/seadas-7.3/ocssw/run/var/anc/2016/240

Cannot create geolocation from /rsclass/fiji_test/aqua_l1a_gom/A2016240174000.L1A_LAC.x.hdf; exiting.

>=====> attempting modis_GEO.py after removing ancillary_data.db ...

Determining required attitude and ephemeris files...

Input file: /rsclass/fiji_test/aqua_l1a_gom/A2016240174000.L1A_LAC.x.hdf
Start time: 2016240174000
End time: 2016240174459

Downloading 'PM1EPHND.P2016240.1200.003' to /Applications/seadas-7.3/ocssw/run/var/anc/2016/240
Connection error, retrying up to 5 more time(s)
Connection error, retrying up to 4 more time(s)
Connection error, retrying up to 3 more time(s)
Connection error, retrying up to 2 more time(s)
Connection error, retrying up to 1 more time(s)
URL attempted: oceandata.sci.gsfc.nasa.gov
Well, this is embarrassing...an error occurred that we just cannot get past...
Here is what we know: ('The read operation timed out',)
Please retry this request at a later time.
*** ERROR: The HTTP transfer failed with status code 500.
*** Please check your network connection and for the existence of the remote file:
*** oceandata.sci.gsfc.nasa.gov/cgi/getfile/PM1EPHND.P2016240.1200.003
***
*** Also check to make sure you have write permissions under the directory:
*** /Applications/seadas-7.3/ocssw/run/var/anc/2016/240

Cannot create geolocation from /rsclass/fiji_test/aqua_l1a_gom/A2016240174000.L1A_LAC.x.hdf; exiting.

.
.
.

>=====>  checking for best ancillary data (Met and Ozone) locally and retrieving from web if needed...

Searching database: /Applications/seadas-7.3/ocssw/run/var/ancillary_data.db

Input file: /rsclass/fiji_test/aqua_l1a_gom/A2016240174000.L1A_LAC.x.hdf
Start time: 2016240174000
End time: 2016240174459

  Found: /Applications/seadas-7.3/ocssw/run/var/anc/2016/240/N201624018_MET_NCEPR2_6h.hdf
  Found: /Applications/seadas-7.3/ocssw/run/var/anc/2016/240/N201624018_MET_NCEPR2_6h.hdf
  Found: /Applications/seadas-7.3/ocssw/run/var/anc/2016/240/N201624012_MET_NCEPR2_6h.hdf
  Found: /Applications/seadas-7.3/ocssw/run/var/anc/2016/240/N201624000_O3_AURAOMI_24h.hdf
  Found: /Applications/seadas-7.3/ocssw/run/var/anc/2016/241/N201624100_O3_AURAOMI_24h.hdf
  Found: /Applications/seadas-7.3/ocssw/run/var/anc/2016/241/N201624100_O3_AURAOMI_24h.hdf
  Found: /Applications/seadas-7.3/ocssw/run/var/anc/2016/240/N2016240_SST_OIV2AV_24h.nc
Downloading 'N201624000_SEAICE_NSIDC_24h.hdf' to /Applications/seadas-7.3/ocssw/run/var/anc/2016/240
Connection error, retrying up to 5 more time(s)
Connection error, retrying up to 4 more time(s)
Connection error, retrying up to 3 more time(s)
Connection error, retrying up to 2 more time(s)
Connection error, retrying up to 1 more time(s)
URL attempted: oceandata.sci.gsfc.nasa.gov
Well, this is embarrassing...an error occurred that we just cannot get past...
Here is what we know: ('The read operation timed out',)
Please retry this request at a later time.
*** ERROR: The HTTP transfer failed with status code 500.
*** Please check your network connection and for the existence of the remote file:
*** oceandata.sci.gsfc.nasa.gov/cgi/getfile/N201624000_SEAICE_NSIDC_24h.hdf
***
*** Also check to make sure you have write permissions under the directory:
*** /Applications/seadas-7.3/ocssw/run/var/anc/2016/240

Bruce,

Got an unhappy backend server (notice the first error was a 404 "File not found").
Once it's back on the playing field, the script will find it and all should work.

Sean

Yep, works great now!! Thanks for making me happy (again).

Bruce

Where / when will the announcement be posted?

Where: On this forum and via the mailing list.
When: When its ready to be announced...

Sean

Hi,

I'm trying the manual install method:

"So, you will need to manually pull down the install_ocssw.py script follow the
manual install/update process (scroll down the page about halfway)"

In the instructions it says:

  $ ./install_ocssw.py --install-dir=$HOME/ocssw --git-branch=<version> --aqua --seawifs
Replace <version> in the --git-branch option with the desired branch. (e.g.--git-branch=v7.2)

I've tried v7.3 and v7.3.2 here as the git=branch. v7.3 works but v7.3.2 does not. If I do v7.3 am I downloading the latest version of SeaDAS or is this older? If older, what do I put here to download the latest version?

Thanks,
Ryan

Yes.  The git repo version only goes to the first "dot",  v7.3 is the latest.

Sean

Ok - great! Thanks for the fast response.

Ryan

I'm have the following problem and can not get the update the ocssw processors or install a clean copy of them.

[] ~/work/oceandata$ wget -N https://oceandata.sci.gsfc.nasa.gov/ocssw/install_ocssw.py
--2017-01-10 13:37:12--  https://oceandata.sci.gsfc.nasa.gov/ocssw/install_ocssw.py
Resolving oceandata.sci.gsfc.nasa.gov (oceandata.sci.gsfc.nasa.gov)... 169.154.128.84, 2001:4d0:2418:128::84
Connecting to oceandata.sci.gsfc.nasa.gov (oceandata.sci.gsfc.nasa.gov)|169.154.128.84|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 25424 (25K) [text/plain]
Saving to: ‘install_ocssw.py’

100%[======================================>] 25,424      --.-K/s   in 0.001s

2017-01-10 13:37:12 (47.5 MB/s) - ‘install_ocssw.py’ saved [25424/25424]

[] ~/work/oceandata$ chmod +x install_ocssw.py
[] ~/work/oceandata$ ./install_ocssw.py -v -i . -g https://oceandata.sci.gsfc.nasa.gov/ocssw/ -b v7.2 -a linux_64 -s --aqua

gitBase     = https://oceandata.sci.gsfc.nasa.gov/ocssw/
gitBranch   = v7.2
install dir = /home/woodbri/work/oceandata
arch        = linux_64

git user.name = "*************"
git user.email = "*****************"
Installing bundles.sha256sum (1 of 14)
Downloading bundles.sha256sum
--2017-01-10 13:38:40--  https://oceandata.sci.gsfc.nasa.gov/ocssw/bundles.sha256sum
Resolving oceandata.sci.gsfc.nasa.gov (oceandata.sci.gsfc.nasa.gov)... 169.154.128.84, 2001:4d0:2418:128::84
Connecting to oceandata.sci.gsfc.nasa.gov (oceandata.sci.gsfc.nasa.gov)|169.154.128.84|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 2578 (2.5K) [text/plain]
Saving to: ‘bundles.sha256sum’

100%[======================================>] 2,578       --.-K/s   in 0s

2017-01-10 13:38:40 (276 MB/s) - ‘bundles.sha256sum’ saved [2578/2578]

Loading checksum file.
Installing common (2 of 14)
Downloading new directory -  /home/woodbri/work/oceandata/run/data/common
Downloading common.bundle
--2017-01-10 13:38:40--  https://oceandata.sci.gsfc.nasa.gov/ocssw/common.bundle
Resolving oceandata.sci.gsfc.nasa.gov (oceandata.sci.gsfc.nasa.gov)... 169.154.128.84, 2001:4d0:2418:128::84
Connecting to oceandata.sci.gsfc.nasa.gov (oceandata.sci.gsfc.nasa.gov)|169.154.128.84|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1270770080 (1.2G) [text/plain]
Saving to: ‘common.bundle’

100%[====================================>] 1,270,770,080 7.09MB/s   in 2m 41s

2017-01-10 13:41:21 (7.55 MB/s) - ‘common.bundle’ saved [1270770080/1270770080]

comparing checksum for common.bundle
Cloning into '/home/woodbri/work/oceandata/run/data/common'...
Receiving objects: 100% (182/182), 1.18 GiB | 35.40 MiB/s, done.
Resolving deltas: 100% (60/60), done.
Checking connectivity... done.
Checking out files: 100% (112/112), done.
Updating (fetch) existing repository - /home/woodbri/work/oceandata/run/data/common
fatal: unable to access 'https://oceandata.sci.gsfc.nasa.gov/ocssw/common.git/': gnutls_handshake() failed: Handshake failed
Error - Could not run "cd /home/woodbri/work/oceandata/run/data/common; git fetch"
[] ~/work/oceandata$ ls
install_ocssw.py  run

You have:
./install_ocssw.py -v -i . -g https://oceandata.sci.gsfc.nasa.gov/ocssw/ -b v7.2 -a linux_64 -s --aqua

I think you need the v7.3 branch for the https changes, but you also have a problem with git:
Updating (fetch) existing repository - /home/woodbri/work/oceandata/run/data/common
fatal: unable to access 'https://oceandata.sci.gsfc.nasa.gov/ocssw/common.git/': gnutls_handshake() failed: Handshake failed
Error - Could not run "cd /home/woodbri/work/oceandata/run/data/common; git fetch"

To get more details, try running:
cd /home/woodbri/work/oceandata/run/data/common
git fetch -v
Git uses libcurl, so it is possible that your supporting libraries (libcurl and the TLS library used by libcurl) are too old for NASA's https servers.  See using curl with new https server.

Thank you for your response and suggestions.

I have tried to install v7.2 and v7.3 with the same results.
As for curl and TLS, I have the following:

[] ~/work/oceandata$ openssl ciphers -tls -v 'HIGH:!ADH:!MD5:@STRENGTH' | egrep "ECDHE-ECDSA-CHACHA20-POLY1305|ECDHE-ECDSA-AES256-GCM-SHA384|ECDHE-ECDSA-AES128-GCM-SHA256|ECDHE-ECDSA-AES256-SHA384|ECDHE-ECDSA-AES128-SHA256"
ECDHE-ECDSA-AES256-GCM-SHA384 TLSv1.2 Kx=ECDH     Au=ECDSA Enc=AESGCM(256) Mac=AEAD
ECDHE-ECDSA-AES256-SHA384 TLSv1.2 Kx=ECDH     Au=ECDSA Enc=AES(256)  Mac=SHA384
ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH     Au=ECDSA Enc=AESGCM(128) Mac=AEAD
ECDHE-ECDSA-AES128-SHA256 TLSv1.2 Kx=ECDH     Au=ECDSA Enc=AES(128)  Mac=SHA256

[] ~/work/oceandata$ /usr/bin/python -c 'import ssl ; print ssl.OPENSSL_VERSION'
OpenSSL 1.0.1f 6 Jan 2014

[] ~/work/oceandata$ gnutls-cli -l | grep TLS1.2
TLS_ANON_DH_AES_128_CBC_SHA256                          0x00, 0x6c      TLS1.2
TLS_ANON_DH_AES_256_CBC_SHA256                          0x00, 0x6d      TLS1.2
TLS_DHE_DSS_AES_128_CBC_SHA256                          0x00, 0x40      TLS1.2
TLS_DHE_DSS_AES_256_CBC_SHA256                          0x00, 0x6a      TLS1.2
TLS_DHE_RSA_AES_128_CBC_SHA256                          0x00, 0x67      TLS1.2
TLS_DHE_RSA_AES_256_CBC_SHA256                          0x00, 0x6b      TLS1.2
TLS_RSA_NULL_SHA256                                     0x00, 0x3b      TLS1.2
TLS_RSA_AES_128_CBC_SHA256                              0x00, 0x3c      TLS1.2
TLS_RSA_AES_256_CBC_SHA256                              0x00, 0x3d      TLS1.2
Protocols: VERS-SSL3.0, VERS-TLS1.0, VERS-TLS1.1, VERS-TLS1.2

[] ~/work/oceandata$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 14.04.5 LTS
Release:        14.04
Codename:       trusty

Ubuntu 14.04 $ gnutls-cli -l | grep TLS1.2 is missing the required ciphers.    For this reason, I upgraded my home systems to Linux Mint 18 (based on Ubuntu 16.04) and get
$ cat /etc/os-release
NAME="Linux Mint"
VERSION="18.1 (Serena)"
ID=linuxmint
ID_LIKE=ubuntu
PRETTY_NAME="Linux Mint 18.1"
VERSION_ID="18.1"
HOME_URL="http://www.linuxmint.com/"
SUPPORT_URL="http://forums.linuxmint.com/"
BUG_REPORT_URL="http://bugs.launchpad.net/linuxmint/"
VERSION_CODENAME=serena
UBUNTU_CODENAME=xenial
$ gnutls-cli --version
gnutls-cli 3.4.10
$ gnutls-cli -l  | egrep "ECDHE_ECDSA_CHACHA20_POLY1305|ECDHE_ECDSA_AES_256_GCM_SHA384|ECDHE_ECDSA_AES_128_GCM_SHA256|ECDHE_ECDSA_AES_256-SHA384|ECDHE_ECDSA_AES_128_SHA256"
TLS_ECDHE_ECDSA_AES_128_GCM_SHA256                  0xc0, 0x2b  TLS1.2
TLS_ECDHE_ECDSA_AES_256_GCM_SHA384                  0xc0, 0x2c  TLS1.2
TLS_ECDHE_ECDSA_CHACHA20_POLY1305                   0xcc, 0xa9  TLS1.2

At work there is a Linux Mint 17.3 system that appears to have working git, but it has been heavily tweaked by someone else so I have no idea how it compares with a "stock" Ubuntu 14.04.  On closer investigation, the report that git was workiong was from Dec. 20th.  The Mint 17.3 system now has the same issue.

@SeanBailey This requirement seems a little severe We are not in a position to upgrade our servers to Ubuntu 16.04 and I have to believe that this impacts a lot of other users also. It looks like this issue is being tracked at https://bugs.launchpad.net/ubuntu/+source/gnutls26/+bug/1444656 but who knows when or if any action will be taken on it since it has been open since 2015-04-15. Is it possible to add any of the chipers supported above to your servers so these older clients can connect?

I have asked sysadmins for some brainstorming on this.  There may or may not be a solution or workaround to be found, but we'll try.
Unfortunately, if we want to keep our servers available we cannot use older ciphers.  To do so puts us in jeopardy of having the lights
turned off on us.

Sean

NASA is not the only organization moving to https or beefing up crypto protocols for https.  Many sites will be adopting the Mozilla Modern recommendations so if there are problems with NASA now there will be problems with others sites in the future.   It would be helpful to have some guidance for end user workarounds, e.g., for sites such as mine where git is failing, can we install a recent linux distro on an (older, unloved, and unwanted) PC, run install_ocssw.py there, and then use rsync to update the .git directories on the much loved trusty [pun intended] LTS machines?

You might also add a note to the gnults bug mentioned above that this is impacting all .gov sites and hopefully they will up the priority on this issue and fix it in TLS where it should probably be dealt with. Meanwhile, sounds like I need to figure out a work around. Thanks all! I'll watch this thread for any updates.

Could you create a static v7.2 and a v7.3 tarball of the git respository that we can download then build it manually? It seems that the only problem we have to in installing the software. Once we get it installed we should be good if we do not need to use git https access.

Would this work?

Hi,

I was also trying to update SeaDAS manually since its not happening with GUI. Its not able to communicate with the server after when it reaches towards the end.

rslab@rslab-pc:~/softwares$ sudo ./install_ocssw.py --install-dir=/home/rslab/seadas-7.3.2/ocssw --git-branch=v7.3 --aqua --seawifs --meris
Installing bundles.sha256sum (1 of 16)
--2017-01-13 20:54:15--  https://oceandata.sci.gsfc.nasa.gov/ocssw/bundles.sha256sum
Resolving oceandata.sci.gsfc.nasa.gov (oceandata.sci.gsfc.nasa.gov)... 169.154.128.84, 2001:4d0:2418:128::84
Connecting to oceandata.sci.gsfc.nasa.gov (oceandata.sci.gsfc.nasa.gov)|169.154.128.84|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 2578 (2.5K) [text/plain]
Saving to: ‘bundles.sha256sum’

bundles.sha256sum   100%[===================>]   2.52K  --.-KB/s    in 0s     

2017-01-13 20:54:16 (295 MB/s) - ‘bundles.sha256sum’ saved [2578/2578]

Installing common (2 of 16)
Installing OCSSW_bash.env (3 of 16)
--2017-01-13 20:54:20--  https://oceandata.sci.gsfc.nasa.gov/ocssw/OCSSW_bash.env
Resolving oceandata.sci.gsfc.nasa.gov (oceandata.sci.gsfc.nasa.gov)... 169.154.128.84, 2001:4d0:2418:128::84
Connecting to oceandata.sci.gsfc.nasa.gov (oceandata.sci.gsfc.nasa.gov)|169.154.128.84|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 2894 (2.8K) [text/plain]
Saving to: ‘OCSSW_bash.env’

OCSSW_bash.env      100%[===================>]   2.83K  --.-KB/s    in 0s     

2017-01-13 20:54:21 (362 MB/s) - ‘OCSSW_bash.env’ saved [2894/2894]

Installing README (4 of 16)
--2017-01-13 20:54:21--  https://oceandata.sci.gsfc.nasa.gov/ocssw/README
Resolving oceandata.sci.gsfc.nasa.gov (oceandata.sci.gsfc.nasa.gov)... 169.154.128.84, 2001:4d0:2418:128::84
Connecting to oceandata.sci.gsfc.nasa.gov (oceandata.sci.gsfc.nasa.gov)|169.154.128.84|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 3358 (3.3K) [text/plain]
Saving to: ‘README’

README              100%[===================>]   3.28K  --.-KB/s    in 0s     

2017-01-13 20:54:22 (418 MB/s) - ‘README’ saved [3358/3358]

Installing ocrvc (5 of 16)
Installing meris (6 of 16)
Installing modis (7 of 16)
Installing modisa (8 of 16)
Installing hmodisa (9 of 16)
Installing seawifs (10 of 16)
Installing bin (11 of 16)
Installing bin3 (12 of 16)
Installing scripts (13 of 16)
Python version acceptable
Installing seawifs-luts (14 of 16)
Installing aqua-luts (15 of 16)
Error downloading MYD02_Emissive_LUTs.V6.1.35.26_OC2.hdf
Could not communicate with the server.
Please check your network connections and try again later.
Error - Could not install luts for aqua

Its getting stuck in the end . I will be really grateful if anyone of you can help with this issue.

Thanking You,
Swathy.

Many open source software providers have moved away from tarballs in favour of direct access to version control repos.  This has important advantages: a) less work for software providers (so they can focus on improving the software), b) users can pull particular versions, and c) lower network costs.  We already have the git bundles, so git can be used to apply recent changes without the need to download a full snapshop of the system.  In our lab I'm hoping we can NFS mount the ocssw directories on a newer (e.g., "Mozilla Modern") system and run install_ocssw.py that way.  I tried this once and all went well until
$ install_ocssw.py -v -b v7.3 --install-dir=/net/<Mint 17 box>/home/<seadas_user>/ocssw.nfs --src --curl --aqua --viirsn
[...]
Checking that an installed executable can run
Checking Python version
Python version acceptable
Installing aqua-luts (14 of 16)
Error downloading MYD02_Emissive_LUTs.V6.1.35.26_OC2.hdf
Could not communicate with the server.
Please check your network connections and try again later.
Error - Could not install luts for aqua


Setting OCSSWROOT to  the NFS directory and manually running update_luts.py worked, but when I go back to the Mint 17 box, getanc.py failed with Error retrieving ancillary file list.  A few minutes later, getanc.py worked.  Still on the Mint 17 box, seadas_benchmark.bash failed with:
===============================================================
Creating L1B file (modis_L1B.py)...
===============================================================

modis_L1B.py A2006167181000.L1A A2006167181000.GEO -o A2006167181000.L1B \
                     --del-hkm --del-qkm
Traceback (most recent call last):
  File "/home/seadas/ocssw.nfs/run/scripts/modis_L1B.py", line 144, in <module>
    sys.exit(main())
  File "/home/seadas/ocssw.nfs/run/scripts/modis_L1B.py", line 137, in main
    modis_env(l1b_instance)
  File "/home/seadas/ocssw.nfs/run/scripts/modules/modis_utils.py", line 398, in modis_env
    '.'.join(self.refl_lut.split('.')[1:5]))
AttributeError: ModisL1B instance has no attribute 'refl_lut'
Command exited with non-zero status 1
so I ran getanc.py -v aqua and the benchmark script worked.

@Swathy -- I just had the same problem so it may have been an internet hiccup.  You can try running update_luts.py manually.   I don't think the install_ocssw.py script does anything more after "Python version acceptable" than run update_luts.py for each requested sensor.

I'm fine with using git over tarballs when I can access git. I have a github account and support multiple projects and have no problem accessing them via Ubuntu 14.04. The problem is that people can not access the oceancolor servers from Ubuntu 14.04 which has an EOL in 2019. And from this thread others seem to be having similar issues so tarballs was a proposal to potentially work around the issue. Hopefully Sean and the IT staff can come up with another idea.

One of our sysadmins took on the challenge.  Below is what he found to work...be forewarned it is not an ideal solution, but it is a workable one.

In short, Git will need to be build from source against the openssl version of libcurl instead of the default gnutls.  The following instructions work on Ubuntu
14.04 and should also work for any other OS (although may need modification for different package management systems - e.g. yum).

$ sudo apt-get install build-essential libcurl4-openssl-dev
$ cd ~
$ wget https://github.com/git/git/archive/v2.11.0.zip -O git_2.11.0.zip
$ unzip git_2.11.0.zip
$ cd git-2.11.0
$ make prefix=~/usr/local all && make prefix=~/usr/local install

NOTE: this locally built version of GIT is installed in ~/usr/local/bin/ and your shell environment PATH should be modified to point to
the local version of Git. For example, export the PATH variable in bash:
$ export PATH=~/usr/local/bin:$PATH

$ which git
/home/$USERNAME/usr/local/bin/git

$ git --version
git version 2.11.0

$ git ls-remote
"https://oceandata.sci.gsfc.nasa.gov/git/obpgpyutils/" HEAD
be40eb6871cf353024c22bdd8e4b014b91bc414a        HEAD

As you can see, Git is able to connect to OBPG NASA and negotiate the SSL Certificate as well as the TLS Cipher.
The HEAD request shows the current hash of the git repository.

Hi,

I have been trying to updated to the https. I downloaded "install_ocssw.py" and entered the following commands

chmod +x install_ocssw.py
./install_ocssw.py --install-dir=$HOME/ocssw --git-branch=v7.3 --aqua --seawifs
./install_ocssw.py --install-dir=$HOME/ocssw --git-branch=v7.3 --src
./install_ocssw.py --install-dir=$HOME/ocssw --direct-broadcast

I had to add "--no-check-certificate" in the wget line inside "install_ocssw.py" ...After that it seem to work perfectly. But something went wrong because the LUT update still looking for an http site instead of https.

This is the error message when trying to process a L1A file ....

>=====>  generating geolocation file from modis L1A standard resolution bands...
** Files utcpole.dat/leapsec.dat are more than 2w old
** Running update_luts to update files..

[ MODIS ]
Connection interrupted, retrying up to 5 more time(s)
Connection interrupted, retrying up to 4 more time(s)
Connection interrupted, retrying up to 3 more time(s)
Connection interrupted, retrying up to 2 more time(s)
Connection error, retrying up to 1 more time(s)
We failed to reach a server.
Please retry this request at a later time.
URL attempted: oceandata.sci.gsfc.nasa.gov
HTTP Error: 301 - Moved Permanently
* ERROR: The download failed with status code: 301
* Please check your network connection and for the existence of the remote file:
* oceandata.sci.gsfc.nasa.gov/Ancillary/LUTs/modis/leapsec.dat

I also try to update the luts manually and got this error.      "./update_luts.py aqua"
URL attempted: oceandata.sci.gsfc.nasa.gov
Well, this is embarrassing...an error occurred that we just cannot get past...
Here is what we know: timed out
Please retry this request at a later time.
* ERROR: The download failed with status code: 301
* Please check your network connection and for the existence of the remote file:
* oceandata.sci.gsfc.nasa.gov/Ancillary/LUTs/modis/leapsec.dat
^CWell, the server did not like this...reports: Moved Permanently
* ERROR: The download failed with status code: 301
* Please check your network connection and for the existence of the remote file:
* oceandata.sci.gsfc.nasa.gov/Ancillary/LUTs/modis/utcpole.dat
Error! could not establish a network connection. Check your network connection.
If you do not find a problem, please try again later.

Any help will be greatly appreciated!

If your $OCSSWROOT/run/scripts directory was successfully updated it should use https connections and not be redirected.
Verify that you see the following when running git log in the script direcotry:

$ cd $OCSSWROOT/run/scripts
$ git log --oneline
d2acbb5 removed non-ASCII character bug
459d226 http/https proxy fixed
373787f added remote set to ensure https switch
5eca43e updated for default to https
bee89a4 updated for connecting via HTTPS

Sean

Running ./install_ocssw.py --install-dir=$HOME/ocssw --direct-broadcast without the --git-branch=v7.3 resets the branch to "master".    When things aren't working smoothly it is a good idea to add the -v flag for verbose output.  You would have seen messages like "Switching to branch -  master".

What do you get from git status -v in the scripts directory?  You should see:

$ cd $OCSSWROOT/run/scripts
$ git status
# On branch v7.3
nothing to commit, working directory clean

I suspect you will have:

$ git status
# On branch master
nothing to commit, working directory clean

If you are still having problems you should mention what platform you are using as there are workarounds for some OS's.  It defeats the purpose of https to use  "--no-check-certificate" with wget, but if you have other issues you need some updates, you may get current certificates once your software is updated.

Ah good catch
The master and v7.3 should be the same, but may need a git pull.
or better yet git checkout v7.3

Sean

Good morning,

Last night Bruce help me fix the wget issue, so now I can use the wget for https. Although, I still have the same errors I mention before. I downloaded the
"install_ocssw.py" and try

chmod +x install_ocssw.py
./install_ocssw.py --install-dir=$HOME/ocssw --git-branch=v7.3 --aqua --seawifs
./install_ocssw.py --install-dir=$HOME/ocssw --git-branch=v7.3 --direct-broadcast

$ git status
On branch v7.3
Your branch is up-to-date with 'origin/v7.3'.
Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git checkout -- <file>..." to discard changes in working directory)

  modified:   install_ocssw.py

Untracked files:
  (use "git add <file>..." to include in what will be committed)

  .DS_Store
  PM1ATTNR.P2016347.0000.003.2016347150040

no changes added to commit (use "git add" and/or "git commit -a")

$git log --oneline
8be1ed9 updated for viirs support
031be9a latest updates for v7.3.2
294880c v7.3.2 updates
dc2aaa5 fix to handle the recent modifications to the oceandata.sci.gsfc.nasa.gov data server
d599556 part of a process to bring the master branch up to date with current release
a430c5a updated scripts for v7.3
86c8d60 added support for running behind an anonymous proxy
99e099e added new utility
27bde04 updates to multilevel processor and related scripts
084ba2e updated install_ocssw.py to use wget if available and skip the stash pop
0086b3b fix for mlp file typer bug
4dd1ecd updated source to V2015.1
d90b0a7 merged in latest V2015.0 updates
2929b05 updated to V2015.0
5e48bae added logic to update calfile entry in msl12_defaults.par file for VIIRS
d0eb862 updating install script and processing rules for mlp
c9b3bfe fix for OSX version parsing
a4ed380 modified modis_L1B_utils.py to select binary based on Mac OS version
abe677a several minor tweaks
0ca5fcb updated to pull xcal data for viirsn
d044d18 updated the httpdl method to reuse open connection
5c77f03 fixed DEFAULT_ANC_DIR_TEXT issue
ad2d7f4 added branch and clean install options
2f4234c added clean install option
07faba7 added resource limit setting
cbb2ad6 added resource limit setting
a86e4ea modified resource limit settings
20fc7f9 updated scripts for v7.1
a29b1eb updated to latest v7.1 versions
982aeff added hico to list of known sensors
7ca9237 updating to latest namer/mlp code
80b407c merged trunk updates for next_level_namer and multi_level_processor'
e0706e5 fix for local option to be truly local - no connection required
22a5202 merged updates to tag
9d9f85f fixes for mlp
107e5cd added files updated for SeaDAS release 7.0.1
e1d1f96 updated to V2013.3
ac41c2c put collection_id info in l1aextract utils to fix bug
23d1adc added binary compatibility check
1ec1b4b mods to add chksum check
a7c618c fix for viirsn lut update
7f99000 added ulimit set
4328298 modified progress meter output
05aba3d fixed verbose bug
cd747b1 added verbose option to httpdl
8ecdd0e fixed aquarius lut update;renamed seadas_processor.py
f3bffa0 updated to latest trunk version
0198eb1 mods for MODIS C6 GEO and L1A
254ac7b fix for git path problem; fix for stdout buffer
4638acb updated to V2013.1
ca45735 updated to latest trunk versions
cb6933f latest trunk updates
da77f59 added new python script
feb8a6e merged in latest OCSSW trunk
a039d0b merged in latest OCSSW trunk
8b872cf updated to latest trunk
977ca41 initial import scripts

Sean, I do not see the line about https ...so it seems I am doing something wrong

I forgot to mention that I am using a Mac OS X El Capitan v10.11.4

I have MacOS El Capitan v 10.11.6 (with the supplemental security update), but your version should work once you have newer openssl support in some python-2.7.   The Apple version pf python-2.7 uses a very old openssl library that does not work with the NASA servers.   Macports or Anaconda python-2.7 are known to work, but homebrew python-2.7 may use the outdated Apple openssl library.

See Topic 6427 for details of the required Python SSL library version and tests.

I have this version of phyton:

2.7.11 |Anaconda custom (x86_64)| (default, Dec  6 2015, 18:57:58)
[GCC 4.2.1 (Apple Inc. build 5577)]

and I check the openSSL and it says

"OpenSSL 1.0.2g 1 March 2016"

I have deleted SeaDAS and I re-installing it to see if that works. Do you think I need to upgrade my python too? I thought this should work ...

Thanks a lot!!

Hello everyone,
I just want to let you know that after re-installing seadas everything is working again!
Thank you very much for all your help!

I almost have it working completely on my Mac (Sierra).  Re-installing SeaDAS did not fix everything.

I installed Anaconda following Sean's recommendation ( https://www.continuum.io/downloads ).  I now have:
python --version: Python 2.7.13 :: Anaconda 4.3.0 (x86_64)
openssl version: OpenSSL 1.0.2k  26 Jan 2017

I reinstalled SeaDAS, installed OCSSW packages from the command line as follows with the latest install_ocssw.py script:
./install_ocssw.py --install-dir=$OCSSWROOT --git-branch=v7.3 --aqua --seawifs

Updating luts from the command line with the update_luts.py script works fine.  I can also create GEO files from the command line with the modis_GEO.py script.

The GUI is still giving me some problems.  When I click on "Update OC Processors" I get an error message "ocssw installation script failed to download. Please check network connection or 'seadas.ocssw.root' variable in the "seadas.config" file. possible cause of error: Operation time out"
Also, when checking for a new SeaDAS version at start up I get the message "The on-line version check failed, an I/O error occurred.  Please note that..."

I am now able to create GEO files using the GUI, earlier it did give me some problems, but they somehow went away (without making any changes to the SeaDAS install)...  Error also regarding network connection, see snippet from log below when it failed earlier.

710     EPHEMERIS RECORDS WRITTEN.  LAST JED =    2458896.50
720     EPHEMERIS RECORDS WRITTEN.  LAST JED =    2459216.50
Error! could not establish a network connection. Check your network connection.
If you do not find a problem, please try again later.
Creating binary planetary ephemeris file...
Determining required attitude and ephemeris files...

Input file: /Users/treble/Downloads/test/A2015349174000.L1A_LAC
Start time: 2015349174000
End time: 2015349174459

Cannot create geolocation from /Users/treble/Downloads/test/A2015349174000.L1A_LAC; exiting.

The GUI update issue will not be resolved until, well, we update the GUI.
Was that your remaining problem?  

Sean

Ah, ok, I see.  I was confused because inymar wrote that 'everything' worked.

I am not sure if the 'Get Ancillary' is working in l2gen when processing a SeaWiFS L2_GAC file.  I just tried it and nothing happens when I click Get Ancillary (any of the options).  In the panel below it still says ANCILLARY INPUTS Default = climatology.  Also, in the OCSSW_LOG_l2gen.txt it seems it is not using 'real time' data:
  met1   = /Applications/seadas-7.3/ocssw/run/data/common/met_climatology_v2014.hdf
  ozone1 = /Applications/seadas-7.3/ocssw/run/data/common/ozone_climatology_v2014.hdf
  no2    = /Applications/seadas-7.3/ocssw/run/data/common/no2_climatology_v2013.hdf

Hmmm...worked for me.  What file were you trying to process and (if you can recall) about when did you attempt (so we can see if there's anything in the logs)

Sean

Strange, I tried a couple just now...
S2003239163104.L1A_GAC
S2000243154434.L1A_GAC

env | grep anc gives: L2GEN_ANC=/Applications/seadas-7.3/ocssw/run/var/anc
ls /Applications/seadas-7.3/ocssw/run/var/anc gives 2015 (so no year 2000 or 2003...).

The log says:

Opening meteorological files.
  met1   = /Applications/seadas-7.3/ocssw/run/data/common/met_climatology_v2014.hdf
  met2   =
  met3   =
  ozone1 = /Applications/seadas-7.3/ocssw/run/data/common/ozone_climatology_v2014.hdf
  ozone2 =
  ozone3 =
  no2    = /Applications/seadas-7.3/ocssw/run/data/common/no2_climatology_v2013.hdf



Well, sitting on my couch (so just like very other SeaDAS user ):

$ getanc.py --verbose S2003239163104.L1A_GAC
Searching database: /Users/Shared/seadas-7.3/ocssw/run/var/ancillary_data.db
Determining pass start and end times...

Input file: S2003239163104.L1A_GAC
Start time: 2003239163103
End time: 2003239171443

Downloading 'N200323918_MET_NCEPR2_6h.hdf' to /Users/Shared/seadas-7.3/ocssw//run/var/anc/2003/239
  Found: /Users/Shared/seadas-7.3/ocssw//run/var/anc/2003/239/N200323918_MET_NCEPR2_6h.hdf
Downloading 'N200323912_MET_NCEPR2_6h.hdf' to /Users/Shared/seadas-7.3/ocssw//run/var/anc/2003/239
Downloading 'N200323900_O3_EPTOMS_24h.hdf' to /Users/Shared/seadas-7.3/ocssw//run/var/anc/2003/239
Downloading 'N200324000_O3_EPTOMS_24h.hdf' to /Users/Shared/seadas-7.3/ocssw//run/var/anc/2003/240
  Found: /Users/Shared/seadas-7.3/ocssw//run/var/anc/2003/240/N200324000_O3_EPTOMS_24h.hdf
Downloading 'N2003239_SST_OIV2AVAM_24h.nc' to /Users/Shared/seadas-7.3/ocssw//run/var/anc/2003/239
Downloading 'N200323900_SEAICE_NSIDC_24h.hdf' to /Users/Shared/seadas-7.3/ocssw//run/var/anc/2003/239

Created 'S2003239163104.L1A_GAC.anc' l2gen parameter text file:

icefile=/Users/Shared/seadas-7.3/ocssw//run/var/anc/2003/239/N200323900_SEAICE_NSIDC_24h.hdf
met1=/Users/Shared/seadas-7.3/ocssw//run/var/anc/2003/239/N200323912_MET_NCEPR2_6h.hdf
met2=/Users/Shared/seadas-7.3/ocssw//run/var/anc/2003/239/N200323918_MET_NCEPR2_6h.hdf
met3=/Users/Shared/seadas-7.3/ocssw//run/var/anc/2003/239/N200323918_MET_NCEPR2_6h.hdf
ozone1=/Users/Shared/seadas-7.3/ocssw//run/var/anc/2003/239/N200323900_O3_EPTOMS_24h.hdf
ozone2=/Users/Shared/seadas-7.3/ocssw//run/var/anc/2003/240/N200324000_O3_EPTOMS_24h.hdf
ozone3=/Users/Shared/seadas-7.3/ocssw//run/var/anc/2003/240/N200324000_O3_EPTOMS_24h.hdf
sstfile=/Users/Shared/seadas-7.3/ocssw//run/var/anc/2003/239/N2003239_SST_OIV2AVAM_24h.nc

- All optimal ancillary data files were determined and downloaded. -


Perhaps there is a network issue when you try?
Try running manually, using --verbose and --refreshDB  to see if you get anything like:
Connection error, retrying up to 5 more time(s)
...


Sean

Thanks for testing...

From the command line getting ancillary data works fine for me, thanks for the reminder.
When I do the following:
From the GUI, start l2gen, and click on "Get Ancillary", I get a pop-up window for a 5-10 seconds with "Retrieving ancillary files", but the window below doesn't get updated, nor is anything downloaded in my L2GEN_ANC directory.
Issuing getanc.py from the command line downloads the files with no problem.
Going back to the GUI and clicking on "Get Ancillary" now works.  The files get listed in the window frame below...

So the downloading only doesn't work from the GUI.  It will use them when they are already downloaded.  Maybe still an http/https thing?

When trying to update SeaDAS using the help provided in the page - https://seadas.gsfc.nasa.gov/installers/ , I downloaded installer script, ran the command - chmod +x install_ocssw.py and while trying to run the installer using this command -
ws135-54:installer madhur$ sudo ./install_ocssw.py --install-dir=/Applications/seadas-7.3.2/ocssw/  --git-branch=7.3 --aqua --seawifs

following error appeared :

ws135-54:installer madhur$ sudo ./install_ocssw.py --install-dir=/Applications/seadas-7.3.2/ocssw/  --git-branch=7.3 --aqua --seawifs
Password:
Installing bundles.sha256sum (1 of 15)
--2017-02-08 16:01:34--  https://oceandata.sci.gsfc.nasa.gov/ocssw/bundles.sha256sum
Resolving oceandata.sci.gsfc.nasa.gov... 169.154.128.84, 2001:4d0:2418:128::84
Connecting to oceandata.sci.gsfc.nasa.gov|169.154.128.84|:443... connected.
Unable to establish SSL connection.
Error - Executing command "cd /Applications/seadas-7.3.2/ocssw; wget --tries=5 --wait=5 https://oceandata.sci.gsfc.nasa.gov/ocssw/bundles.sha256sum"
Bundle checksum file (bundles.sha256sum) not downloaded
ws135-54:installer madhur$

My OS X El Capitan version 10.11.6

I am not verse in terminal and mac OS. So could you help me to diagnose the problem and help me to solve it?

Thanks,
Madhur

that is odd, as my test worked both from the GUI and the command line...I'm a wee bit stumped...

Couple of things...

The  branch includes the 'v' prefix, so the --git-branch option should be "--git-branch=v7.3"

You'll also need an alternate version of python as the Apple provided version does not support SSL in the manner required to make a connection to our servers.
see: https://oceancolor.gsfc.nasa.gov/forum/oceancolor/topic_show.pl?pid=27730

Sean

@madhur: see my post above on the steps to take.  You probably only have to install Anaconda, with python 2.7:

"
I installed Anaconda following Sean's recommendation ( https://www.continuum.io/downloads ).  I now have:
python --version: Python 2.7.13 :: Anaconda 4.3.0 (x86_64)
openssl version: OpenSSL 1.0.2k  26 Jan 2017

I reinstalled SeaDAS, installed OCSSW packages from the command line as follows with the latest install_ocssw.py script:
./install_ocssw.py --install-dir=$OCSSWROOT --git-branch=v7.3 --aqua --seawifs

Updating luts from the command line with the update_luts.py script works fine.  I can also create GEO files from the command line with the modis_GEO.py script.
"