
The forum is locked.
The Ocean Color Forum has transitioned over to the Earthdata Forum (https://forum.earthdata.nasa.gov/). The information existing below will be retained for historical reference. Please sign into the Earthdata Forum for active user support.
wget --post-data="subID=528&cksum=1" -q -O - https://oceandata.sci.gsfc.nasa.gov/api/file_search.cgi
wget --post-data="subID=528&cksum=1" -q -O - https://oceandata.sci.gsfc.nasa.gov/api/file_search.cgi > search.cksums && awk '{print $2}' search.cksums | wget --auth-no-challenge=on --noverbose --no-clobber --base="https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/" --load-cookies ~/.urs_cookies --save-cookies ~/.urs_cookies --keep-session-cookies --input-file - && sha1sum -c search.cksums
product the following output
3765810dd7c33912924f3aec5aa0ae916af59710 A2020135083500.L1A_LAC.bz2
8d65b5471cf98c7af51c20e0823eb5d603e2aaba A2020135163000.L1A_LAC.bz2
1dcf679a103819acaecf39f0feca7e4a7a76ef5d A2020135180500.L1A_LAC.bz2
41797784600f48afdb58ac96fed67684971590ce A2020135181000.L1A_LAC.bz2
5daf1f26119e63edcca8aed50a06332557f82534 A2020136074000.L1A_LAC.bz2
0dca87f5715ec0a32b1549f458bcc94533aa0370 A2020136153500.L1A_LAC.bz2
90e7fced017a2d2ace0030fa1e0e87917d063277 A2020136171000.L1A_LAC.bz2
780e2b593a377410343c5252a5be280f2d1b23fc A2020137161500.L1A_LAC.bz2
65a07a1f749df9e21f695d814f980a41ec8a2b62 A2020137175500.L1A_LAC.bz2
20 redirections exceeded.
20 redirections exceeded.
20 redirections exceeded.
20 redirections exceeded.
20 redirections exceeded.
20 redirections exceeded.
20 redirections exceeded.
20 redirections exceeded.
20 redirections exceeded.
with great regularity and repeatability. I have it on good authority that nothing has changed with our network.
They have worked since the requirement to do the whole authorization stuff.
Bruce
You're not the first to report such a behavior
Nothing changed on our end either...the trouble is, there is a third party involved. We're still investigating what might have changed. As soon as we come up with some information, I'll relay it.
Sean

I've had frequent error "47 Too many redirects." using curl (V7.29.0) ever since you activated authentication. Here is a quick figure with the number of errors per day. It is not normalised to the number of attempted downloads. The number of attempted downloads is pretty constant, though. So this gives a good indication when things have been changing.
Regards, Stefan
PS: The line is a 7 day running mean.

We are investigating this on our end, but have yet to be able to replicate the problem.
We are also planning to implement an alternative authentication mechanism for non-interactive downloads with the goal of minimizing the need to re-authenticate within a session.
Sean
curl: (47) Maximum (50) redirects followed
I will turn up verbosity and see if I can get more info.
Stefan
35 - SSL connect error. The SSL handshaking failed.
51 - The peer's SSL certificate or SSH MD5 fingerprint was not OK.
56 - Failure in receiving network data.
They are not of concern but might help solving the problem.
Stefan
I have a more detailed log. I don't want to post the log here as I don't know whether it contains sensitive information. What is the best way to get it to you?
Stefan
We cannot d/l anything now. I have many failures (on separate campuses) as do others working in various locations.
See attached file.
The troubling part is
* Re-using existing connection! (#0) with host oceandata.sci.gsfc.nasa.gov
* Connected to oceandata.sci.gsfc.nasa.gov (169.154.128.84) port 443 (#0)
* Server auth using Basic with user 'oo_lab'
> GET /ob/getfile/restrict?code=e60913d028b88e6b82aa14c2e3c4f8cb21a249c9cd180fabbac498fa40da301e HTTP/1.1
> Authorization: Basic b29fbGFiOlpYdzN5eXdDN295RQ==
> User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.44 zlib/1.2.3 libidn/1.18 libssh2/1.4.2
> Host: oceandata.sci.gsfc.nasa.gov
> Accept: */*
> Cookie: app-obdaac=2792e4f92c8bc81104c284b64a22f87a7124cdc4
>
< HTTP/1.1 403 Forbidden
< Server: nginx
< Date: Thu, 21 May 2020 23:39:03 GMT
< Content-Type: text/html; charset=utf-8
< Content-Length: 665
< Connection: keep-alive
< Keep-Alive: timeout=60
< ETag: "585d4968-299"
<
john
Trying direct download through the browser (chrome), redirection return error:
.:. ERROR .:.
OceanColor Biology Processing Group (OBPG)
Sorry, an error has occurred. Use the back button to return to the previous page or go to the Ocean Color Home Page.
Maybe the maintenance explains the errors but if not, I am adding the above hoping that it will be useful.
[bmurch@optics0 ~]$ ll /shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/138/PM1ATTNR.P2020138.0???.003
-rw-rw-r-- 1 bmurch cms_optics 665 May 22 00:13 /shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/138/PM1ATTNR.P2020138.0200.003
-rw-rw-r-- 1 bmurch cms_optics 665 May 22 00:13 /shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/138/PM1ATTNR.P2020138.0400.003
-rw-rw-r-- 1 bmurch cms_optics 665 May 22 00:13 /shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/138/PM1ATTNR.P2020138.0600.003
-rw-rw-r-- 1 bmurch cms_optics 665 May 22 00:13 /shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/138/PM1ATTNR.P2020138.0800.003
/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/138/PM1ATTNR.P2020138.0200.003: HTML document text
/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/138/PM1ATTNR.P2020138.0400.003: HTML document text
/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/138/PM1ATTNR.P2020138.0600.003: HTML document text
/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/138/PM1ATTNR.P2020138.0800.003: HTML document text
/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/138/PM1ATTNR.P2020138.1000.003: HTML document text
/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/138/PM1ATTNR.P2020138.1200.003: HTML document text
/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/138/PM1ATTNR.P2020138.1400.003: HTML document text
/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/138/PM1ATTNR.P2020138.1600.003: data
/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/138/PM1ATTNR.P2020138.1800.003: HTML document text
/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/138/PM1ATTNR.P2020138.2000.003: HTML document text
/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/138/PM1ATTNR.P2020138.2200.003: HTML document text
A minor change to how we connect to them has been put in place and seems to have fixed the issue...wondering if it fixes the previous max redirects error as well.
Try your downloads again and let me know if it's still an issue for you.
Sean
B

Guess we still have some forensics to do...
< HTTP/1.1 302 Found
< Server: nginx
< Date: Fri, 22 May 2020 15:25:33 GMT
< Transfer-Encoding: chunked
< Connection: keep-alive
< Keep-Alive: timeout=60
< Location: https://urs.earthdata.nasa.gov/oauth/authorize?client_id=Z0u-MdLNypXBjiDREZ3roA&response_type=code&redirect_uri=https%3A%2F%2Foceandata.sci.gsfc.nasa.gov%2Fob%2Fgetfile%2Frestrict
< Expires: Mon, 01 Jan 1970 00:00:00 GMT
< Cache-Control: no-cache, must-revalidate, max-age=0, no-store
< Pragma: no-cache
* Replaced cookie app-obdaac="880e0858d5c227ef26f66a3c8c0344c38f0c9875" for domain oceandata.sci.gsfc.nasa.gov, path /, expire 0
< Set-Cookie: app-obdaac=880e0858d5c227ef26f66a3c8c0344c38f0c9875; path=/; secure
< Referrer-Policy: no-referrer
< Expect-CT: max-age=31536000, enforce
< Strict-Transport-Security: max-age=31536000; includeSubDomains; preload
< Content-Security-Policy: upgrade-insecure-requests; default-src 'self' oceancolor.gsfc.nasa.gov data:; script-src 'self' 'unsafe-inline' 'unsafe-eval' www.google-analytics.com www.googletagmanager.com cdn.earthdata.nasa.gov dap.digitalgov.gov data:; style-src 'self' 'unsafe-inline' code.jquery.com cdn.earthdata.nasa.gov; img-src 'self' data: oceancolor.gsfc.nasa.gov www.google-analytics.com cdn.earthdata.nasa.gov
<
* Ignoring the response-body
{ [data not shown]
0 0 0 0 0 0 0 0 --:--:-- 0:00:06 --:--:-- 0* Connection #0 to host oceandata.sci.gsfc.nasa.gov left intact
* Issue another request to this URL: 'https://urs.earthdata.nasa.gov/oauth/authorize?client_id=Z0u-MdLNypXBjiDREZ3roA&response_type=code&redirect_uri=https%3A%2F%2Foceandata.sci.gsfc.nasa.gov%2Fob%2Fgetfile%2Frestrict'
* Re-using existing connection! (#1) with host urs.earthdata.nasa.gov
* Connected to urs.earthdata.nasa.gov (2001:4d0:241a:4081::89) port 443 (#1)
* Server auth using Basic with user 'oo_processing'
> GET /oauth/authorize?client_id=Z0u-MdLNypXBjiDREZ3roA&response_type=code&redirect_uri=https%3A%2F%2Foceandata.sci.gsfc.nasa.gov%2Fob%2Fgetfile%2Frestrict HTTP/1.1
> Authorization: Basic b29fcHJvY2Vzc2luZzpJZzJpd2tzRkpVaHY=
> User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.19.1 Basic ECC zlib/1.2.11 libidn/1.18 libssh2/1.4.2
> Host: urs.earthdata.nasa.gov
> Accept: */*
> Cookie: _urs-gui_session=470e1acb2262074a485b8519e88c629a; urs_user_already_logged=yes
>
< HTTP/1.1 302 Found
< Server: nginx/1.17.5
< Date: Fri, 22 May 2020 15:25:33 GMT
< Content-Type: text/html; charset=utf-8
< Transfer-Encoding: chunked
< Connection: keep-alive
< X-Frame-Options: SAMEORIGIN
< X-XSS-Protection: 1; mode=block
< X-Content-Type-Options: nosniff
< X-Download-Options: noopen
< X-Permitted-Cross-Domain-Policies: none
< Referrer-Policy: strict-origin-when-cross-origin
< Access-Control-Allow-Origin: null
< Access-Control-Allow-Credentials: true
< Access-Control-Allow-Methods: GET, POST
< Access-Control-Expose-Headers: true
< Location: https://oceandata.sci.gsfc.nasa.gov/ob/getfile/restrict?code=37602330dd13da46df4ff6d4fd0dff6aff91e3694c68e428c649a2b9d6ee5173
< Cache-Control: no-cache
* Replaced cookie urs_user_already_logged="yes" for domain earthdata.nasa.gov, path /, expire 1590247533
< Set-Cookie: urs_user_already_logged=yes; domain=earthdata.nasa.gov; path=/; expires=Sat, 23 May 2020 15:25:33 GMT
* Replaced cookie _urs-gui_session="470e1acb2262074a485b8519e88c629a" for domain urs.earthdata.nasa.gov, path /, expire 1590247533
< Set-Cookie: _urs-gui_session=470e1acb2262074a485b8519e88c629a; path=/; expires=Sat, 23 May 2020 15:25:33 GMT; HttpOnly
< X-Request-Id: 8efb344a-907d-4803-9410-70715f4df3ff
< X-Runtime: 0.098615
< Strict-Transport-Security: max-age=31536000
<
* Ignoring the response-body
{ [data not shown]
191 191 0 191 0 0 28 0 --:--:-- 0:00:06 --:--:-- 28* Connection #1 to host urs.earthdata.nasa.gov left intact
* Maximum (50) redirects followed
curl: (47) Maximum (50) redirects followed
* Closing connection #0
* Closing connection #1
$curl_command_returns:
I create and different .urs_cookies file for each run I do to avoid the issue.
So for instance, today I may create one that is .urs_cookies_viirs_052620 and run something by hand.
Or, because I have several scripts that run that may overlap ( ex. running on different interfaces (machines) but the same user and file system, so the name of the script is used e.g. from my log:
STATUS: $cookiejar: /optics/home/oo_processing/.urs_cookies_get_viirs_h5_gsfc
The file is created at the start of a request and deleted at the end.
I wish it were this simple

Problem
You encounter this error message when running the curl command
curl -u admin:changeme -L -k -X GET --header "Accept: application/json" https:///rest/v1.0/projects
The output was: curl: (47) Maximum (50) redirects followed
Solution
This is because the WebServer Loadbalancer VIP was configured to forward the request only to http port 80 of EF webserver. Then in /apache/conf, the EF webserver redirects all requests on port 80 to port 443 of the Loadbalancer.
This goes in a loop and fails after 50 retries.
You can resolve this issue by configuring the WebServer Loadbalancer VIP to forward the request to https port 443 of the EF webservers that it is load balancing.
https://support.cloudbees.com/hc/en-us/articles/360032824192-KBEC-00402-Resolve-curl-47-Maximum-50-redirects-followed-issue
The issue is more likely related to the cookie, as deleting it resolved the problem for Bruce....until it cropped up again, but still likely a cookie issue. The Earthdata login folks are looking more closely to make sure there isn't an issue on their end. The cookie is intended to be for a given session, so keeping it around between sessions isn't necessary and may be problematic.
Sean
I delete my cookies every time. If its in a script, it checks to see if the cookie file exists. If it does, its deleted prior to the curl call.
I never reuse them after the failure, and it fails every time on a list, so I delete the ~/.cookie_jar_00 for instance, and it is created new with the following command:
If I try to run the command again, using the same cookie file, it starts the redirect failures immediately without downloading any files, so I MUST delete the cookie file.
curl -b ~/.cookie_jar_00 -c ~/.cookie_jar_00 -L -n --interface 2607:fe50:0:6330::100 --retry 5 --retry-delay 2 --max-time 0 --remote-name-all https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/{$(sed ':a;N;$!ba;s/\n/,/g' /shares/cms_optics/virtual_ant/S4P/bin/fa_density/non_fai_rois/CAPE_COD/x00)}
Where the bolded x00 section is a list of about 950 files to download. When it get to 47, it dies with the redirect error.
I can delete the ~/.cookie_jar_00, and remove the files already downloaded from the list, and start with the next files in the x00 list file.
It will fail again at the 47 file d/l mark with the new cookie file ~/.cookie_jar_00 that was created at the start of the run.
I have not seen the cookie file ~/.cookie_jar_00 change (after creation until failure).
It bears the time stamp of the original creation when the command is run.
Thus, I think its unlikely that its on the cookie side locally.
Having said that, if Earthdata has one load balance pointed to port 80 in error somewhere, and that doesn't exist (as you say), then it will fail like described, I think?
Brock
I cannot get your cURL command to work (appropriately sanitized for my Mac - something not happy with the sed bit), but this one does work for me:
curl -d "sensor=octs&sdate=1996-11-01&edate=1997-01-01&dtype=L3b&addurl=1&results_as_file=1&search=*DAY_CHL*" https://oceandata.sci.gsfc.nasa.gov/api/file_search |grep getfile | cut -d "'" -f 2 | xargs -n 1 curl -LJO -n -c ~/.urs_cookies -b ~/.urs_cookies
It only returns 61 files, but does return them without issue. As I don't get a redirect failure I cannot explain why you seem to consistently fail after 47 files.
Others with the problem seems to suggest it IS a cookie issue.
The Earthdata folks live under the same network/IT constraints we do, so I seriously doubt they've got an issue with their load balancing - again, that error would not be intermittent but would deny ALL traffic.
Sean
It pops up on your forums and its what others here, see too.
Hopefully it will be tracked down at some point.
Cheers,
Stay safe!
Powered by mwForum 2.29.7 © 1999-2015 Markus Wichitill