Starting Saturday morning, these 2 commands
wget --post-data="subID=528&cksum=1" -q -O - https://oceandata.sci.gsfc.nasa.gov/api/file_search.cgi

wget --post-data="subID=528&cksum=1" -q -O - https://oceandata.sci.gsfc.nasa.gov/api/file_search.cgi > search.cksums && awk '{print $2}' search.cksums | wget --auth-no-challenge=on --noverbose --no-clobber --base="https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/"  --load-cookies ~/.urs_cookies --save-cookies ~/.urs_cookies --keep-session-cookies --input-file - && sha1sum -c search.cksums

product the following output

3765810dd7c33912924f3aec5aa0ae916af59710  A2020135083500.L1A_LAC.bz2
8d65b5471cf98c7af51c20e0823eb5d603e2aaba  A2020135163000.L1A_LAC.bz2
1dcf679a103819acaecf39f0feca7e4a7a76ef5d  A2020135180500.L1A_LAC.bz2
41797784600f48afdb58ac96fed67684971590ce  A2020135181000.L1A_LAC.bz2
5daf1f26119e63edcca8aed50a06332557f82534  A2020136074000.L1A_LAC.bz2
0dca87f5715ec0a32b1549f458bcc94533aa0370  A2020136153500.L1A_LAC.bz2
90e7fced017a2d2ace0030fa1e0e87917d063277  A2020136171000.L1A_LAC.bz2
780e2b593a377410343c5252a5be280f2d1b23fc  A2020137161500.L1A_LAC.bz2
65a07a1f749df9e21f695d814f980a41ec8a2b62  A2020137175500.L1A_LAC.bz2
20 redirections exceeded.
20 redirections exceeded.
20 redirections exceeded.
20 redirections exceeded.
20 redirections exceeded.
20 redirections exceeded.
20 redirections exceeded.
20 redirections exceeded.
20 redirections exceeded.

with great regularity and repeatability.  I have it on good authority that nothing has changed with our network.
They have worked since the requirement to do the whole authorization stuff.

Bruce

Bruce,

You're not the first to report such a behavior
Nothing changed on our end either...the trouble is, there is a third party involved.   We're still investigating what might have changed.  As soon as we come up with some information, I'll relay it.

Sean

Not that it probably really matters...  It actually goes back to last Thursday (5/14) morning

Oh, it matters   That timing jives with other "oddities" we've experienced - and points more at that third party...

Hi Sean,
I've had frequent error "47 Too many redirects." using curl (V7.29.0) ever since you activated authentication. Here is a quick figure with the number of errors per day. It is not normalised to the number of attempted downloads. The number of attempted downloads is pretty constant, though. So this gives a good indication when things have been changing.
Regards, Stefan

PS: The line is a 7 day running mean.

Thanks.  This is a bit disturbing...if you have any log information that could provide some insight into this, we'd welcome it!
We are investigating this on our end, but have yet to be able to replicate the problem. 
We are also planning to implement an alternative authentication mechanism for non-interactive downloads with the goal of minimizing the need to re-authenticate within a session.

Sean

All I have is:
curl: (47) Maximum (50) redirects followed
I will turn up verbosity and see if I can get more info.
Stefan

... other errors I get only occasionally are:
35 - SSL connect error. The SSL handshaking failed.
51 - The peer's SSL certificate or SSH MD5 fingerprint was not OK.
56 - Failure in receiving network data.
They are not of concern but might help solving the problem.
Stefan

Sean,
I have a more detailed log. I don't want to post the log here as I don't know whether it contains sensitive information. What is the best way to get it to you?
Stefan

Sean,

We cannot d/l anything now. I have many failures (on separate campuses) as do others working in various locations.
See attached file.

The troubling part is
* Re-using existing connection! (#0) with host oceandata.sci.gsfc.nasa.gov                                                                                                                                                                 
* Connected to oceandata.sci.gsfc.nasa.gov (169.154.128.84) port 443 (#0)                                                                                                                                                                  
* Server auth using Basic with user 'oo_lab'                                                                                                                                                                                               

> GET /ob/getfile/restrict?code=e60913d028b88e6b82aa14c2e3c4f8cb21a249c9cd180fabbac498fa40da301e HTTP/1.1                                                                                                                                  
> Authorization: Basic b29fbGFiOlpYdzN5eXdDN295RQ==                                                                                                                                                                                        
> User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.44 zlib/1.2.3 libidn/1.18 libssh2/1.4.2                                                                                                                           
> Host: oceandata.sci.gsfc.nasa.gov                                                                                                                                                                                                        
> Accept: */*                                                                                                                                                                                                                              
> Cookie: app-obdaac=2792e4f92c8bc81104c284b64a22f87a7124cdc4                                                                                                                                                                              
>                                                                                                                                                                                                                                          

< HTTP/1.1 403 Forbidden                                                                                                                                                                                                                   
< Server: nginx                                                                                                                                                                                                                            
< Date: Thu, 21 May 2020 23:39:03 GMT                                                                                                                                                                                                      
< Content-Type: text/html; charset=utf-8                                                                                                                                                                                                   
< Content-Length: 665                                                                                                                                                                                                                      
< Connection: keep-alive                                                                                                                                                                                                                   
< Keep-Alive: timeout=60                                                                                                                                                                                                                   
< ETag: "585d4968-299"                                                                                                                                                                                                                     
<

There was supposed to be network maintenance going on between 17:00-00:00 EDT that could cause intermittent connectivity problems.

john

Just an additional information

Trying direct download through the browser (chrome), redirection return error:
              .:. ERROR .:.
OceanColor Biology Processing Group (OBPG)
Sorry, an error has occurred. Use the back button to return to the previous page or go to the Ocean Color Home Page.

Maybe the maintenance explains the errors but if not, I am adding the above hoping that it will be useful.

Note that the problem exists for anc data as well
[bmurch@optics0 ~]$ ll /shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/138/PM1ATTNR.P2020138.0???.003
-rw-rw-r-- 1 bmurch cms_optics 665 May 22 00:13 /shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/138/PM1ATTNR.P2020138.0200.003
-rw-rw-r-- 1 bmurch cms_optics 665 May 22 00:13 /shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/138/PM1ATTNR.P2020138.0400.003
-rw-rw-r-- 1 bmurch cms_optics 665 May 22 00:13 /shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/138/PM1ATTNR.P2020138.0600.003
-rw-rw-r-- 1 bmurch cms_optics 665 May 22 00:13 /shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/138/PM1ATTNR.P2020138.0800.003

/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/138/PM1ATTNR.P2020138.0200.003:     HTML document text                                 
/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/138/PM1ATTNR.P2020138.0400.003:      HTML document text                                 
/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/138/PM1ATTNR.P2020138.0600.003:      HTML document text                                 
/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/138/PM1ATTNR.P2020138.0800.003:     HTML document text                                 
/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/138/PM1ATTNR.P2020138.1000.003:      HTML document text                                 
/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/138/PM1ATTNR.P2020138.1200.003:      HTML document text                                 
/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/138/PM1ATTNR.P2020138.1400.003:      HTML document text                                 
/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/138/PM1ATTNR.P2020138.1600.003:      data                                               
/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/138/PM1ATTNR.P2020138.1800.003:     HTML document text                                 
/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/138/PM1ATTNR.P2020138.2000.003:      HTML document text                                 
/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/138/PM1ATTNR.P2020138.2200.003:      HTML document text

I don't have confirmation that the issue was related to the maintenance done on the authentication servers...but the problem did seem to start about that time...
A minor change to how we connect to them has been put in place and seems to have fixed the issue...wondering if it fixes the previous max redirects error as well.

Try your downloads again and let me know if it's still an issue for you.

Sean

Well I just tried my automated download and I still get redirection problems...

B

...but not a 403 error?

Just the redirection error (but I never got a 403 error, so from my perspective, no change)

Well,  at least the situation didn't get worse
Guess we still have some forensics to do...

For whatever it's worth our downloads seem to be back to normal starting about 13:00 UTC (aka 9 EDT) today

I have the same problem with redirections.

< HTTP/1.1 302 Found                                                                                                                                                              
< Server: nginx                                                                                                                                                                   
< Date: Fri, 22 May 2020 15:25:33 GMT                                                                                                                                             
< Transfer-Encoding: chunked                                                                                                                                                      
< Connection: keep-alive                                                                                                                                                          
< Keep-Alive: timeout=60                                                                                                                                                          
< Location: https://urs.earthdata.nasa.gov/oauth/authorize?client_id=Z0u-MdLNypXBjiDREZ3roA&response_type=code&redirect_uri=https%3A%2F%2Foceandata.sci.gsfc.nasa.gov%2Fob%2Fgetfile%2Frestrict                                                                                                                                                                      
< Expires: Mon, 01 Jan 1970 00:00:00 GMT                                                                                                                                          
< Cache-Control: no-cache, must-revalidate, max-age=0, no-store                                                                                                                   
< Pragma: no-cache                                                                                                                                                                
* Replaced cookie app-obdaac="880e0858d5c227ef26f66a3c8c0344c38f0c9875" for domain oceandata.sci.gsfc.nasa.gov, path /, expire 0                                                  
< Set-Cookie: app-obdaac=880e0858d5c227ef26f66a3c8c0344c38f0c9875; path=/; secure                                                                                                 
< Referrer-Policy: no-referrer                                                                                                                                                    
< Expect-CT: max-age=31536000, enforce                                                                                                                                            
< Strict-Transport-Security: max-age=31536000; includeSubDomains; preload                                                                                                         
< Content-Security-Policy: upgrade-insecure-requests; default-src 'self' oceancolor.gsfc.nasa.gov data:; script-src 'self' 'unsafe-inline' 'unsafe-eval' www.google-analytics.com www.googletagmanager.com cdn.earthdata.nasa.gov dap.digitalgov.gov data:; style-src 'self' 'unsafe-inline' code.jquery.com cdn.earthdata.nasa.gov; img-src 'self' data: oceancolor.gsfc.nasa.gov www.google-analytics.com cdn.earthdata.nasa.gov                                                                                                                      
<                                                                                                                                                                                 
* Ignoring the response-body                                                                                                                                                      
{ [data not shown]                                                                                                                                                                
  0     0    0     0    0     0      0      0 --:--:--  0:00:06 --:--:--     0* Connection #0 to host oceandata.sci.gsfc.nasa.gov left intact                                     
* Issue another request to this URL: 'https://urs.earthdata.nasa.gov/oauth/authorize?client_id=Z0u-MdLNypXBjiDREZ3roA&response_type=code&redirect_uri=https%3A%2F%2Foceandata.sci.gsfc.nasa.gov%2Fob%2Fgetfile%2Frestrict'                                                                                                                                           
* Re-using existing connection! (#1) with host urs.earthdata.nasa.gov
* Connected to urs.earthdata.nasa.gov (2001:4d0:241a:4081::89) port 443 (#1)
* Server auth using Basic with user 'oo_processing'

> GET /oauth/authorize?client_id=Z0u-MdLNypXBjiDREZ3roA&response_type=code&redirect_uri=https%3A%2F%2Foceandata.sci.gsfc.nasa.gov%2Fob%2Fgetfile%2Frestrict HTTP/1.1
> Authorization: Basic b29fcHJvY2Vzc2luZzpJZzJpd2tzRkpVaHY=
> User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.19.1 Basic ECC zlib/1.2.11 libidn/1.18 libssh2/1.4.2
> Host: urs.earthdata.nasa.gov
> Accept: */*
> Cookie: _urs-gui_session=470e1acb2262074a485b8519e88c629a; urs_user_already_logged=yes
>

< HTTP/1.1 302 Found
< Server: nginx/1.17.5
< Date: Fri, 22 May 2020 15:25:33 GMT
< Content-Type: text/html; charset=utf-8
< Transfer-Encoding: chunked
< Connection: keep-alive
< X-Frame-Options: SAMEORIGIN
< X-XSS-Protection: 1; mode=block
< X-Content-Type-Options: nosniff
< X-Download-Options: noopen
< X-Permitted-Cross-Domain-Policies: none
< Referrer-Policy: strict-origin-when-cross-origin
< Access-Control-Allow-Origin: null
< Access-Control-Allow-Credentials: true
< Access-Control-Allow-Methods: GET, POST
< Access-Control-Expose-Headers: true
< Location: https://oceandata.sci.gsfc.nasa.gov/ob/getfile/restrict?code=37602330dd13da46df4ff6d4fd0dff6aff91e3694c68e428c649a2b9d6ee5173
< Cache-Control: no-cache
* Replaced cookie urs_user_already_logged="yes" for domain earthdata.nasa.gov, path /, expire 1590247533
< Set-Cookie: urs_user_already_logged=yes; domain=earthdata.nasa.gov; path=/; expires=Sat, 23 May 2020 15:25:33 GMT
* Replaced cookie _urs-gui_session="470e1acb2262074a485b8519e88c629a" for domain urs.earthdata.nasa.gov, path /, expire 1590247533
< Set-Cookie: _urs-gui_session=470e1acb2262074a485b8519e88c629a; path=/; expires=Sat, 23 May 2020 15:25:33 GMT; HttpOnly
< X-Request-Id: 8efb344a-907d-4803-9410-70715f4df3ff
< X-Runtime: 0.098615
< Strict-Transport-Security: max-age=31536000
<
* Ignoring the response-body
{ [data not shown]
191   191    0   191    0     0     28      0 --:--:--  0:00:06 --:--:--    28* Connection #1 to host urs.earthdata.nasa.gov left intact
* Maximum (50) redirects followed

curl: (47) Maximum (50) redirects followed
* Closing connection #0
* Closing connection #1
$curl_command_returns:

Could it be this simple?
https://support.cloudbees.com/hc/en-us/articles/360032824192-KBEC-00402-Resolve-curl-47-Maximum-50-redirects-followed-issue

We just went through a troubleshooting session where the redirect issue was solved after the user deleted the old cookie file and was able to successfully execute the command with a new cookie file was generated. It is hopefully that simple for everyone else. Please give it a try.

Well, for me, nuking the .urs_cookies file seems to have solved the problem.

Bruce,
I create and different .urs_cookies file for each run I do to avoid the issue.
So for instance, today I may create one that is .urs_cookies_viirs_052620 and run something by hand.

Or, because I have several scripts that run that may overlap ( ex. running on different interfaces (machines) but the same user and file system, so the name of the script is used e.g. from my log:

STATUS: $cookiejar: /optics/home/oo_processing/.urs_cookies_get_viirs_h5_gsfc

The file is created at the start of a request and deleted at the end.
I wish it were this simple

Well it worked - once.  My overnight job failed again with too many redirections.  Looks like it's time to delete the file on each invocation.

Well, I didn't see a response to this, but since your backend is nginx (a load balancer) this may apply in some way:

Problem
You encounter this error message when running the curl command
curl -u admin:changeme -L -k -X GET --header "Accept: application/json" https:///rest/v1.0/projects
The output was: curl: (47) Maximum (50) redirects followed

Solution
This is because the WebServer Loadbalancer VIP was configured to forward the request only to http port 80 of EF webserver. Then in /apache/conf, the EF webserver redirects all requests on port 80 to port 443 of the Loadbalancer.
This goes in a loop and fails after 50 retries.
You can resolve this issue by configuring the WebServer Loadbalancer VIP to forward the request to https port 443 of the EF webservers that it is load balancing.

https://support.cloudbees.com/hc/en-us/articles/360032824192-KBEC-00402-Resolve-curl-47-Maximum-50-redirects-followed-issue

Not likely.  Such a misconfiguration would prevent ALL downloads.  I can't speak to the configuration of the Earthdata Login set up, but ours does not support port 80 at all, so all traffic is over 443.
The issue is more likely related to the cookie, as deleting it resolved the problem for Bruce....until it cropped up again, but still likely a cookie issue.  The Earthdata login folks are looking more closely to make sure there isn't an issue on their end.  The cookie is intended to be for a given session, so keeping it around between sessions isn't necessary and may be problematic. 

Sean

Sean,

I delete my cookies every time. If its in a script, it checks to see if the cookie file exists. If it does, its deleted prior to the curl call.
I never reuse them after the failure, and it fails every time on a list, so I delete the ~/.cookie_jar_00 for instance, and it is created new with the following command:
If I try to run the command again, using the same cookie file, it starts the redirect failures immediately without downloading any files, so I MUST delete the cookie file.

curl -b ~/.cookie_jar_00 -c ~/.cookie_jar_00 -L -n --interface 2607:fe50:0:6330::100 --retry 5 --retry-delay 2 --max-time 0 --remote-name-all https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/{$(sed ':a;N;$!ba;s/\n/,/g' /shares/cms_optics/virtual_ant/S4P/bin/fa_density/non_fai_rois/CAPE_COD/x00)}

Where the bolded x00 section is a list of about 950 files to download. When it get to 47, it dies with the redirect error.
I can delete the ~/.cookie_jar_00, and remove the files already downloaded from the list, and start with the next files in the x00 list file.
It will fail again at the 47 file d/l mark with the new cookie file ~/.cookie_jar_00 that was created at the start of the run.
I have not seen the cookie file ~/.cookie_jar_00 change (after creation until failure).
It bears the time stamp of the original creation when the command is run.

Thus, I think its unlikely that its on the cookie side locally.

Having said that, if Earthdata has one load balance pointed to port 80 in error somewhere, and that doesn't exist (as you say), then it will fail like described, I think?

Brock

Brock,

I cannot get your cURL command to work (appropriately sanitized for my Mac - something not happy with the sed bit), but this one does work for me:

curl -d "sensor=octs&sdate=1996-11-01&edate=1997-01-01&dtype=L3b&addurl=1&results_as_file=1&search=*DAY_CHL*" https://oceandata.sci.gsfc.nasa.gov/api/file_search |grep getfile | cut -d "'" -f 2 | xargs -n 1 curl -LJO -n -c ~/.urs_cookies -b ~/.urs_cookies

It only returns 61 files, but does return them without issue.  As I don't get a redirect failure I cannot explain why you seem to consistently fail after 47 files.
Others with the problem seems to suggest it IS a cookie issue. 

The Earthdata folks live under the same network/IT constraints we do, so I seriously doubt they've got an issue with their load balancing - again, that error would not be intermittent but would deny ALL traffic.

Sean

I guess we'll have to see. But, I know that I'm not alone in the magic 47 number.
It pops up on your forums and its what others here, see too.

Hopefully it will be tracked down at some point.

Cheers,
Stay safe!

47 is the error number cURL assigns to the max redirect error.