In the past few days, attempts to retrieve ancillary files from oceandata.sci.gsfc.nasa.gov via wget on our server is.sci.gsfc.nasa.gov have started failing due to the connection timing out.  We have also confirmed that we are unable to ping oceandata.sci.gsfc.nasa.gov from is.sci.gsfc.nasa.gov, but we can both ping and retrieve files from oceandata.sci from other machines.  Could a firewall setting be blocking connection attempts from is.sci, or any other ideas why we are unable to connect to oceandata.sci?

Thanks,
Bob Kannenberg/Outreach Coordinator, Direct Readout Laboratory
NASA/Goddard Space Flight Center
Code 606.3, bldg 28, rm W189
Greenbelt, MD 20771
http://directreadout.sci.gsfc.nasa.gov/
https://www.facebook.com/directreadout/

From the system administrator:

Yes. 169.154.128.59 and 2001:4d0:2418:128::59 were blocked.
The ips are now allowed in.
According to the web logs the ips generated getfile errors
which triggered the automatic block. The request URL is valid
but the client looks to be opening and then closing the
connection abruptly causing the 499 error code.

Tommy

Dear Dr Tommy / Sean

I am managing the operation of the MODIS DB ground station at ROPME in Kuwait. We experience some stuck in the automatic processing of the received MODIS-Terra data. Our internet browsers also are no longer able to connect to your new https connection at oceancolor and ocean data portals. Therefore we are not able to get necessary ancillary files to proceed with processing of Terra data. This stuck has started since switching of NASA to the https connection. We have tried to change our local scripts to tune with this change and also tried to check ssl, security certificates, etc but none of these trials led to any solution. We didn't experience any problem so far with processing of MODIS Aqua. I suspect that our IP has been blocked and that we have a similar case to the case of rkannenb at this forum: https://oceancolor.gsfc.nasa.gov/forum/oceancolor/topic_show.pl?tid=6437

Could you please check and advise about possible reasons of this stuck and possible solutions?. Your reply to this will be highly appreciated.
our IP address is: 62.215.132.254

Best wishes.

Wahid

Wahid,

We've been experiencing problems resulting from asymmetric network routes for several weeks, and some sites have experienced SSL issues since our government-mandated switch to https.  I was able to successfully traceroute to your IP this morning, and have seen connection attempts from your site in our firewall logs. If you could send us the output from a traceroute to oceancolor.gsfc.nasa.gov and try a "wget -d https://oceancolor.gsfc.nasa.gov/" we can try to identify whether it is a routing issue or a SSL issue.

Paul Smith

Dear Paul Smith

Thanks a lot for your response. Herein the inquired tracerouting info:

[apex@localhost ~]$ traceroute oceancolor.gsfc.nasa.gov
traceroute to oceancolor.gsfc.nasa.gov (169.154.128.44), 30 hops max, 60 byte packets
1  192.168.0.1 (192.168.0.1)  8.824 ms  8.993 ms  8.809 ms
2  62.215.132.251 (62.215.132.251)  10.922 ms  11.007 ms  10.963 ms
3  172.25.18.81 (172.25.18.81)  12.589 ms *  12.638 ms
4  skb-ace-10g62VL401.fasttelco.net (62.215.2.90)  12.761 ms  12.659 ms  12.704 ms
5  kdc2-10g000-x-skb-ace.fasttelco.net (62.215.2.62)  12.762 ms  12.780 ms  13.207 ms
6  ix-pos-2-1-1.core1.JSD-Jeddah.as6453.net (195.219.167.12)  40.106 ms  43.684 ms  43.662 ms
7  if-xe-11-1-1-50.tcore2.WYN-Marseille.as6453.net (80.231.200.62)  99.274 ms  99.401 ms  99.279 ms
8  if-ae-2-2.tcore1.WYN-Marseille.as6453.net (80.231.217.1)  99.404 ms  99.423 ms  99.482 ms
9  if-ae-8-1600.tcore1.PYE-Paris.as6453.net (80.231.217.6)  98.482 ms  98.440 ms  98.567 ms
10  if-ae-11-2.tcore1.PVU-Paris.as6453.net (80.231.153.49)  98.702 ms  99.206 ms  99.219 ms
11  80.231.153.66 (80.231.153.66)  130.260 ms  130.201 ms  130.299 ms
12  * * *
13  SIMS-INC.ear2.Washington1.Level3.net (4.31.71.130)  1321.139 ms  1320.505 ms  1320.567 ms
14  138.193.103.253 (138.193.103.253)  1303.351 ms  1303.374 ms  1303.343 ms
15  138.193.113.42 (138.193.113.42)  1303.889 ms  1303.886 ms  1303.445 ms
16  198.116.4.90 (198.116.4.90)  1312.497 ms  1312.525 ms  1312.438 ms
17  rtr-s28-hecn-test.sci.gsfc.nasa.gov (169.154.192.153)  1441.055 ms  1441.159 ms  1440.992 ms
18  * * *
19  * * *
20  * * *
21  * * *
22  * * *
23  * * *
24  * * *
25  * * *
26  * * *
27  * * *
28  * * *
29  * * *
30  * * *

C:\Users\Wahid>tracert -4 oceancolor.gsfc.nasa.gov
Tracing route to oceancolor.sci.gsfc.nasa.gov [169.154.128.44]
over a maximum of 30 hops:

  1     1 ms     1 ms     3 ms  192.168.0.1
  2     2 ms     2 ms     2 ms  62.215.132.251
  3    10 ms     3 ms     3 ms  172.25.18.81
  4     6 ms     3 ms     3 ms  skb-ace-10g62VL401.fasttelco.net [62.215.2.90]
  5     5 ms     5 ms     7 ms  kdc2-10g000-x-skb-ace.fasttelco.net [62.215.2.62]
  6    33 ms    33 ms    34 ms  ix-pos-2-1-1.core1.JSD-Jeddah.as6453.net [195.219.167.12]
  7    91 ms   105 ms    94 ms  if-xe-11-1-1-50.tcore2.WYN-Marseille.as6453.net [80.231.200.62]
  8    97 ms   145 ms    93 ms  if-ae-2-2.tcore1.WYN-Marseille.as6453.net [80.231.217.1]
  9    89 ms    94 ms    94 ms  if-ae-8-1600.tcore1.PYE-Paris.as6453.net [80.231.217.6]
10    91 ms    92 ms    89 ms  if-ae-11-2.tcore1.PVU-Paris.as6453.net [80.231.153.49]
11   122 ms   122 ms   122 ms  80.231.153.66
12     *        *      193 ms  ae-2-3601.ear2.Washington1.Level3.net [4.69.206.73]
13     *     1612 ms  1469 ms  SIMS-INC.ear2.Washington1.Level3.net [4.31.71.130]
14  1340 ms  1206 ms  1351 ms  138.193.103.253
15  1268 ms  1247 ms  1358 ms  138.193.113.42
16  1407 ms  1443 ms  1455 ms  198.116.4.90
17  1246 ms  1205 ms  1250 ms  rtr-s28-hecn-test.sci.gsfc.nasa.gov [169.154.192.153]
18  1321 ms  1386 ms  1399 ms  oceancolor.sci.gsfc.nasa.gov [169.154.128.44]

Trace complete.

[apex@localhost ~]$ wget -d https://oceancolor.gsfc.nasa.gov/
DEBUG output created by Wget 1.13.4 on linux-gnu.

URI encoding = `UTF-8'
--2017-01-12 05:53:56--  https://oceancolor.gsfc.nasa.gov/
Resolving oceancolor.gsfc.nasa.gov (oceancolor.gsfc.nasa.gov)... 169.154.128.44, 2001:4d0:2418:128::44
Caching oceancolor.gsfc.nasa.gov => 169.154.128.44 2001:4d0:2418:128::44
Connecting to oceancolor.gsfc.nasa.gov (oceancolor.gsfc.nasa.gov)|169.154.128.44|:443... connected.
Created socket 3.
Releasing 0x08e97cb8 (new refcount 1).
Initiating SSL handshake.
SSL handshake failed.
Closed fd 3
Unable to establish SSL connection.


I will be highly indebted if you can advise how to deal with this stuck.
Looking forward to hearing from you.
Best wishes.

Wahid

Wget 1.13.4 is quite old, so there is a good chance it was built an old version of openssl or gnutls that lacks (Mozilla Modern) crypto support required with many sites that enforce https. The two most common crypto libraries on linux are openssl and gnutls, so it might be useful to check that ciphers required by NASA are supported, e.g.,:

$ gnutls-cli -l  | egrep "ECDHE_ECDSA_CHACHA20_POLY1305|ECDHE_ECDSA_AES_256_GCM_SHA384|ECDHE_ECDSA_AES_128_GCM_SHA256|ECDHE_ECDSA_AES_256-SHA384|ECDHE_ECDSA_AES_128_SHA256"
TLS_ECDHE_ECDSA_AES_128_GCM_SHA256                  0xc0, 0x2b  TLS1.2
TLS_ECDHE_ECDSA_AES_256_GCM_SHA384                  0xc0, 0x2c  TLS1.2
TLS_ECDHE_ECDSA_CHACHA20_POLY1305                   0xcc, 0xa9  TLS1.2

$ openssl ciphers -tls -v 'HIGH:!ADH:!MD5:@STRENGTH' | egrep "ECDHE-ECDSA-CHACHA20-POLY1305|ECDHE-ECDSA-AES256-GCM-SHA384|ECDHE-ECDSA-AES128-GCM-SHA256|ECDHE-ECDSA-AES256-SHA384|ECDHE-ECDSA-AES128-SHA256"
ECDHE-ECDSA-AES256-GCM-SHA384 TLSv1.2 Kx=ECDH     Au=ECDSA Enc=AESGCM(256) Mac=AEAD
ECDHE-ECDSA-AES256-SHA384 TLSv1.2 Kx=ECDH     Au=ECDSA Enc=AES(256)  Mac=SHA384
ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH     Au=ECDSA Enc=AESGCM(128) Mac=AEAD
ECDHE-ECDSA-AES128-SHA256 TLSv1.2 Kx=ECDH     Au=ECDSA Enc=AES(128)  Mac=SHA256

Wahid,

Thanks for the traceroute and wget information.  Comparing that to traceroutes from us to you shows a definite asymmetry.  I'll pass that on to our ISP for diagnosis.

The recent response concerning old versions of wget is a good point. We've had several cases where older versions of wget do not support the ciphers or certificates that our site does. Unfortunately the older ciphers have known weaknesses that we need to avoid.

As an alternative, if you have access to the curl utility, you could try "curl -IL --verbose https://oceancolor.gsfc.nasa.gov" as a debug tool. It may give you more information about the SSL connection.

Paul

Wahid,

Our ISP checked the routing and found that it is not the cause of the connection problem. That shifts the focus into establishing the SSL connection. Perhaps a newer version of wget, linked against the openssl library, will be successful. It comes down to what ciphers a given version of the application supports. You might also try the curl utility if it is available. If you are successful in reaching us via a current web browser from that system, that eliminates all of the network as the cause of the problem. I realize that the switch to https has caused difficulties for a number of sites

Paul