Ocean Color Forum
The forum is locked.
The Ocean Color Forum has transitioned over to the Earthdata Forum (https://forum.earthdata.nasa.gov/). The information existing below will be retained for historical reference. Please sign into the Earthdata Forum for active user support.
Topic Products and Algorithms / Satellite Data Access / problems with MODIS / VIIRS Data access some anc data d/ls fine
Lo all,
Do you have problems going on with various data (MODIS / VIIRS) Data access?
I can't seem to get files across several installations home/saint petersburg/tampa but I can get some ancillary data like N2021020_SST_OIV2AV_24h.nc?
I can get this:
$ curl -b ~/.urs_cookies -c ~/.urs_cookies -L -n --retry 5 --retry-delay 2 --max-time 0 --output ./N2021020_SST_OIV2AV_24h.nc https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/N2021020_SST_OIV2AV_24h.nc
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
100 1619k 100 1619k 0 0 708k 0 0:00:02 0:00:02 --:--:-- 2890k
But not this:
$ curl -b .urs_cookies_single -c .urs_cookies_single -L -n --output ./PM1EPHND_NRT.A2021020.1555.06 https://oceandata.sci.gsfc.nasa.gov/cgi/PM1EPHND_NRT.A2021020.1555.06
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 665 100 665 0 0 960 0 --:--:-- --:--:-- --:--:-- 959
Or any MODIS/VIIRS
$ curl -b ~/.urs_cookies -c ~/.urs_cookies -L -n --retry 5 --retry-delay 2 --max-time 0 --output ./MOD00.P2021021.0725_1.PDS.bz2 https://oceandata.sci.gsfc.nasa.gov/cgi/MOD00.P2021021.0725_1.PDS.bz2
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 665 100 665 0 0 1205 0 --:--:-- --:--:-- --:--:-- 1206
$ curl -b ~/.urs_cookies -c ~/.urs_cookies -L -n --retry 5 --retry-delay 2 --max-time 0 --output ./MOD00.A2021020.1850_1.PDS.bz2 https://oceandata.sci.gsfc.nasa.gov/cgi/MOD00.A2021020.1850_1.PDS.bz2
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 665 100 665 0 0 1203 0 --:--:-- --:--:-- --:--:-- 1202
$ curl -b .urs_cookies_single -c .urs_cookies_single -L -n --output ./V2021020045400.L1A_SNPP.nc https://oceandata.sci.gsfc.nasa.gov/cgi/V2021020045400.L1A_SNPP.nc
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 665 100 665 0 0 1180 0 --:--:-- --:--:-- --:--:-- 1181
$ file *
MOD00.A2021020.1850_1.PDS.bz2: HTML document, ASCII text, with very long lines
MOD00.P2021021.0725_1.PDS.bz2: HTML document, ASCII text, with very long lines
N2021020_SST_OIV2AV_24h.nc: bzip2 compressed data, block size = 100k
PM1EPHND_NRT.A2021020.1555.06: HTML document, ASCII text, with very long lines
V2021020045400.L1A_SNPP.nc: HTML document, ASCII text, with very long lines
$ bunzip2 N2021020_SST_OIV2AV_24h.nc
bunzip2: Can't guess original name for N2021020_SST_OIV2AV_24h.nc -- using N2021020_SST_OIV2AV_24h.nc.out
$ mv N2021020_SST_OIV2AV_24h.nc.out N2021020_SST_OIV2AV_24h.nc
$ file N2021020_SST_OIV2AV_24h.nc
N2021020_SST_OIV2AV_24h.nc: Hierarchical Data Format (version 5) data
Do you have problems going on with various data (MODIS / VIIRS) Data access?
I can't seem to get files across several installations home/saint petersburg/tampa but I can get some ancillary data like N2021020_SST_OIV2AV_24h.nc?
I can get this:
$ curl -b ~/.urs_cookies -c ~/.urs_cookies -L -n --retry 5 --retry-delay 2 --max-time 0 --output ./N2021020_SST_OIV2AV_24h.nc https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/N2021020_SST_OIV2AV_24h.nc
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
100 1619k 100 1619k 0 0 708k 0 0:00:02 0:00:02 --:--:-- 2890k
But not this:
$ curl -b .urs_cookies_single -c .urs_cookies_single -L -n --output ./PM1EPHND_NRT.A2021020.1555.06 https://oceandata.sci.gsfc.nasa.gov/cgi/PM1EPHND_NRT.A2021020.1555.06
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 665 100 665 0 0 960 0 --:--:-- --:--:-- --:--:-- 959
Or any MODIS/VIIRS
$ curl -b ~/.urs_cookies -c ~/.urs_cookies -L -n --retry 5 --retry-delay 2 --max-time 0 --output ./MOD00.P2021021.0725_1.PDS.bz2 https://oceandata.sci.gsfc.nasa.gov/cgi/MOD00.P2021021.0725_1.PDS.bz2
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 665 100 665 0 0 1205 0 --:--:-- --:--:-- --:--:-- 1206
$ curl -b ~/.urs_cookies -c ~/.urs_cookies -L -n --retry 5 --retry-delay 2 --max-time 0 --output ./MOD00.A2021020.1850_1.PDS.bz2 https://oceandata.sci.gsfc.nasa.gov/cgi/MOD00.A2021020.1850_1.PDS.bz2
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 665 100 665 0 0 1203 0 --:--:-- --:--:-- --:--:-- 1202
$ curl -b .urs_cookies_single -c .urs_cookies_single -L -n --output ./V2021020045400.L1A_SNPP.nc https://oceandata.sci.gsfc.nasa.gov/cgi/V2021020045400.L1A_SNPP.nc
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 665 100 665 0 0 1180 0 --:--:-- --:--:-- --:--:-- 1181
$ file *
MOD00.A2021020.1850_1.PDS.bz2: HTML document, ASCII text, with very long lines
MOD00.P2021021.0725_1.PDS.bz2: HTML document, ASCII text, with very long lines
N2021020_SST_OIV2AV_24h.nc: bzip2 compressed data, block size = 100k
PM1EPHND_NRT.A2021020.1555.06: HTML document, ASCII text, with very long lines
V2021020045400.L1A_SNPP.nc: HTML document, ASCII text, with very long lines
$ bunzip2 N2021020_SST_OIV2AV_24h.nc
bunzip2: Can't guess original name for N2021020_SST_OIV2AV_24h.nc -- using N2021020_SST_OIV2AV_24h.nc.out
$ mv N2021020_SST_OIV2AV_24h.nc.out N2021020_SST_OIV2AV_24h.nc
$ file N2021020_SST_OIV2AV_24h.nc
N2021020_SST_OIV2AV_24h.nc: Hierarchical Data Format (version 5) data
The URLs on your failed examples seem to be missing the /getfile/ element of the URL. The first successful case has it .../cgi/getfile/name-of-file but the others have .../cgi/name-of-file
john
john
John,
I think it was a copy paste thing again but here not on the command line again:
From home worked twice:
[bmurch@dell8 test]$ curl -b ./.urs_cookies_get_viirs -c ./.urs_cookies_get_viirs -L -n --retry 5 --retry-delay 2 --max-time 0 --output ./V2020153180600.L1A_SNPP.nc https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/V2020153180600.L1A_SNPP.nc
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
100 451M 100 451M 0 0 24.6M 0 0:00:18 0:00:18 --:--:-- 29.6M
[bmurch@dell8 test]$ curl -b ~/.urs_cookies -c ~/.urs_cookies -L -n --retry 5 --retry-delay 2 --max-time 0 --output ./MOD00.P2021021.0725_1.PDS.bz2 https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/MOD00.P2021021.0725_1.PDS.bz2
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
100 71.0M 100 71.0M 0 0 15.0M 0 0:00:04 0:00:04 --:--:-- 25.2M
But not this one:
[bmurch@dell8 test]$ curl -b ~/.urs_cookies -c ~/.urs_cookies -L -n --retry 5 --retry-delay 2 --max-time 0 --output ./MOD00.A2021021.0835_1.PDS.bz2 https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/MOD00.A2021021.0835_1.PDS.bz2
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 665 100 665 0 0 16 0 0:00:41 0:00:41 --:--:-- 151
Warning: Transient problem: HTTP error Will retry in 2 seconds. 5 retries
Warning: left.
Throwing away 665 bytes
....
Same files from work (131.247.138.46)
curl -b ~/.urs_cookies -c ~/.urs_cookies -L -n --retry 5 --retry-delay 2 --max-time 0 --output ./MOD00.P2021021.0725_1.PDS.bz2 https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/MOD00.P2021021.0725_1.PDS.bz2
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 665 100 665 0 0 16 0 0:00:41 0:00:41 --:--:-- 208
Warning: Transient problem: HTTP error Will retry in 2 seconds. 5 retries
Warning: left.
....
curl -b ./.urs_cookies_get_viirs -c ./.urs_cookies_get_viirs -L -n --retry 5 --retry-delay 2 --max-time 0 --output ./V2020153180600.L1A_SNPP.nc https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/V2020153180600.L1A_SNPP.nc
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 665 100 665 0 0 16 0 0:00:41 0:00:41 --:--:-- 156
Warning: Transient problem: HTTP error Will retry in 2 seconds. 5 retries
Warning: left.
Throwing away 665 bytes
....
From work 2607:fe50:0:6330::103:
These are coming down now as 665 (and 0 length until they arrive as 665 size html)
-rw-rw-r-- 1 bmurch cms_optics 665 Jan 21 11:38 /cms_zfs/sat_data/modis/l0/2021/021/MOD00.A2021021.0835_1.PDS.bz2
-rw-rw-r-- 1 bmurch cms_optics 665 Jan 21 11:34 /cms_zfs/sat_data/modis/l0/2021/021/MOD00.A2021021.0850_1.PDS.bz2
-rw-rw-r-- 1 bmurch cms_optics 0 Jan 21 11:42 /cms_zfs/sat_data/modis/l0/2021/021/MOD00.A2021021.0930_1.PDS.bz2
-rw-rw-r-- 1 bmurch cms_optics 665 Jan 20 21:00 /cms_zfs/sat_data/modis/l0/2021/021/MOD00.P2021021.0010_1.PDS.bz2
I think it was a copy paste thing again but here not on the command line again:
From home worked twice:
[bmurch@dell8 test]$ curl -b ./.urs_cookies_get_viirs -c ./.urs_cookies_get_viirs -L -n --retry 5 --retry-delay 2 --max-time 0 --output ./V2020153180600.L1A_SNPP.nc https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/V2020153180600.L1A_SNPP.nc
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
100 451M 100 451M 0 0 24.6M 0 0:00:18 0:00:18 --:--:-- 29.6M
[bmurch@dell8 test]$ curl -b ~/.urs_cookies -c ~/.urs_cookies -L -n --retry 5 --retry-delay 2 --max-time 0 --output ./MOD00.P2021021.0725_1.PDS.bz2 https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/MOD00.P2021021.0725_1.PDS.bz2
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
100 71.0M 100 71.0M 0 0 15.0M 0 0:00:04 0:00:04 --:--:-- 25.2M
But not this one:
[bmurch@dell8 test]$ curl -b ~/.urs_cookies -c ~/.urs_cookies -L -n --retry 5 --retry-delay 2 --max-time 0 --output ./MOD00.A2021021.0835_1.PDS.bz2 https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/MOD00.A2021021.0835_1.PDS.bz2
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 665 100 665 0 0 16 0 0:00:41 0:00:41 --:--:-- 151
Warning: Transient problem: HTTP error Will retry in 2 seconds. 5 retries
Warning: left.
Throwing away 665 bytes
....
Same files from work (131.247.138.46)
curl -b ~/.urs_cookies -c ~/.urs_cookies -L -n --retry 5 --retry-delay 2 --max-time 0 --output ./MOD00.P2021021.0725_1.PDS.bz2 https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/MOD00.P2021021.0725_1.PDS.bz2
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 665 100 665 0 0 16 0 0:00:41 0:00:41 --:--:-- 208
Warning: Transient problem: HTTP error Will retry in 2 seconds. 5 retries
Warning: left.
....
curl -b ./.urs_cookies_get_viirs -c ./.urs_cookies_get_viirs -L -n --retry 5 --retry-delay 2 --max-time 0 --output ./V2020153180600.L1A_SNPP.nc https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/V2020153180600.L1A_SNPP.nc
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 665 100 665 0 0 16 0 0:00:41 0:00:41 --:--:-- 156
Warning: Transient problem: HTTP error Will retry in 2 seconds. 5 retries
Warning: left.
Throwing away 665 bytes
....
From work 2607:fe50:0:6330::103:
These are coming down now as 665 (and 0 length until they arrive as 665 size html)
-rw-rw-r-- 1 bmurch cms_optics 665 Jan 21 11:38 /cms_zfs/sat_data/modis/l0/2021/021/MOD00.A2021021.0835_1.PDS.bz2
-rw-rw-r-- 1 bmurch cms_optics 665 Jan 21 11:34 /cms_zfs/sat_data/modis/l0/2021/021/MOD00.A2021021.0850_1.PDS.bz2
-rw-rw-r-- 1 bmurch cms_optics 0 Jan 21 11:42 /cms_zfs/sat_data/modis/l0/2021/021/MOD00.A2021021.0930_1.PDS.bz2
-rw-rw-r-- 1 bmurch cms_optics 665 Jan 20 21:00 /cms_zfs/sat_data/modis/l0/2021/021/MOD00.P2021021.0010_1.PDS.bz2
Can you view the HTML in a browser to see what it is asking for, such as maybe an Earthdata login prompt? Verify .netrc values? Remove the ./.urs_cookies_get_* files to force a new one?
I can't replicate the problem, so the getfile service is functioning. You're saying it is sporadically failing on two different machines, or always working on one, failing on another?
john
I can't replicate the problem, so the getfile service is functioning. You're saying it is sporadically failing on two different machines, or always working on one, failing on another?
john
It might be IPv6 as well:
Here I try with -4 and it works
[bmurch@optics0 ~]$ curl -4 -b ~/.urs_cookies -c ~/.urs_cookies -L -n --retry 5 --retry-delay 2 --max-time 0 --output ./MOD00.A2021021.0835_1.PDS.bz2 https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/MOD00.A2021021.0835_1.PDS.bz2
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 291M 100 291M 0 0 61.4M 0 0:00:04 0:00:04 --:--:-- 87.3M
[bmurch@optics0 ~]$ rm ./MOD00.A2021021.0835_1.PDS.bz2
Here I try with -6 and it fails:
[bmurch@optics0 ~]$ curl -6 -b ~/.urs_cookies -c ~/.urs_cookies -L -n --retry 5 --retry-delay 2 --max-time 0 --output ./MOD00.A2021021.0835_1.PDS.bz2 https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/MOD00.A2021021.0835_1.PDS.bz2
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
110 665 110 665 0 0 901 0 --:--:-- --:--:-- --:--:-- 4650
Warning: Transient problem: HTTP error Will retry in 2 seconds. 5 retries
Warning: left.
^C
[bmurch@optics0 ~]$ rm ./MOD00.A2021021.0835_1.PDS.bz2
Here I try again with -4 and it works again:
[bmurch@optics0 ~]$ curl -4 -b ~/.urs_cookies -c ~/.urs_cookies -L -n --retry 5 --retry-delay 2 --max-time 0 --output ./MOD00.A2021021.0835_1.PDS.bz2 https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/MOD00.A2021021.0835_1.PDS.bz2
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 291M 100 291M 0 0 59.7M 0 0:00:04 0:00:04 --:--:-- 80.8M
[bmurch@optics0 ~]$ rm ./MOD00.A2021021.0835_1.PDS.bz2
Here I try again with -6 and it fails:
[bmurch@optics0 ~]$ curl -6 -b ~/.urs_cookies -c ~/.urs_cookies -L -n --retry 5 --retry-delay 2 --max-time 0 --output ./MOD00.A2021021.0835_1.PDS.bz2 https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/MOD00.A2021021.0835_1.PDS.bz2
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
110 665 110 665 0 0 498 0 0:00:01 0:00:01 --:--:-- 1324
Warning: Transient problem: HTTP error Will retry in 2 seconds. 5 retries
Warning: left.
Throwing away 665 bytes
0 665 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0^C
And fails with the interface specified as well:
$ curl --interface 2607:fe50:0:6330::102 -b ~/.urs_cookies -c ~/.urs_cookies -L -n --retry 5 --retry-delay 2 --max-time 0 --output ./MOD00.A2021021.0835_1.PDS.bz2 https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/MOD00.A2021021.0835_1.PDS.bz2
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
110 665 110 665 0 0 799 0 --:--:-- --:--:-- --:--:-- 2309
Warning: Transient problem: HTTP error Will retry in 2 seconds. 5 retries
Warning: left.
^C
Here I try with -4 and it works
[bmurch@optics0 ~]$ curl -4 -b ~/.urs_cookies -c ~/.urs_cookies -L -n --retry 5 --retry-delay 2 --max-time 0 --output ./MOD00.A2021021.0835_1.PDS.bz2 https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/MOD00.A2021021.0835_1.PDS.bz2
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 291M 100 291M 0 0 61.4M 0 0:00:04 0:00:04 --:--:-- 87.3M
[bmurch@optics0 ~]$ rm ./MOD00.A2021021.0835_1.PDS.bz2
Here I try with -6 and it fails:
[bmurch@optics0 ~]$ curl -6 -b ~/.urs_cookies -c ~/.urs_cookies -L -n --retry 5 --retry-delay 2 --max-time 0 --output ./MOD00.A2021021.0835_1.PDS.bz2 https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/MOD00.A2021021.0835_1.PDS.bz2
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
110 665 110 665 0 0 901 0 --:--:-- --:--:-- --:--:-- 4650
Warning: Transient problem: HTTP error Will retry in 2 seconds. 5 retries
Warning: left.
^C
[bmurch@optics0 ~]$ rm ./MOD00.A2021021.0835_1.PDS.bz2
Here I try again with -4 and it works again:
[bmurch@optics0 ~]$ curl -4 -b ~/.urs_cookies -c ~/.urs_cookies -L -n --retry 5 --retry-delay 2 --max-time 0 --output ./MOD00.A2021021.0835_1.PDS.bz2 https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/MOD00.A2021021.0835_1.PDS.bz2
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 291M 100 291M 0 0 59.7M 0 0:00:04 0:00:04 --:--:-- 80.8M
[bmurch@optics0 ~]$ rm ./MOD00.A2021021.0835_1.PDS.bz2
Here I try again with -6 and it fails:
[bmurch@optics0 ~]$ curl -6 -b ~/.urs_cookies -c ~/.urs_cookies -L -n --retry 5 --retry-delay 2 --max-time 0 --output ./MOD00.A2021021.0835_1.PDS.bz2 https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/MOD00.A2021021.0835_1.PDS.bz2
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
110 665 110 665 0 0 498 0 0:00:01 0:00:01 --:--:-- 1324
Warning: Transient problem: HTTP error Will retry in 2 seconds. 5 retries
Warning: left.
Throwing away 665 bytes
0 665 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0^C
And fails with the interface specified as well:
$ curl --interface 2607:fe50:0:6330::102 -b ~/.urs_cookies -c ~/.urs_cookies -L -n --retry 5 --retry-delay 2 --max-time 0 --output ./MOD00.A2021021.0835_1.PDS.bz2 https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/MOD00.A2021021.0835_1.PDS.bz2
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
110 665 110 665 0 0 799 0 --:--:-- --:--:-- --:--:-- 2309
Warning: Transient problem: HTTP error Will retry in 2 seconds. 5 retries
Warning: left.
^C
I was having issues on all machines with sporadic downloads. I just tried again, and it worked:
[bmurch@optics0 ~]$ curl -4 -b ~/.urs_cookies -c ~/.urs_cookies -L -n --retry 5 --retry-delay 2 --max-time 0 --output ./MOD00.A2021021.0835_1.PDS.bz2 https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/MOD00.A2021021.0835_1.PDS.bz2 % Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 291M 100 291M 0 0 63.4M 0 0:00:04 0:00:04 --:--:-- 83.6M
[bmurch@optics0 ~]$ rm ./MOD00.A2021021.0835_1.PDS.bz2
[bmurch@optics0 ~]$ curl -6 -b ~/.urs_cookies -c ~/.urs_cookies -L -n --retry 5 --retry-delay 2 --max-time 0 --output ./MOD00.A2021021.0835_1.PDS.bz2 https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/MOD00.A2021021.0835_1.PDS.bz2
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 291M 100 291M 0 0 51.6M 0 0:00:05 0:00:05 --:--:-- 65.1M
[bmurch@optics0 ~]$ curl -4 -b ~/.urs_cookies -c ~/.urs_cookies -L -n --retry 5 --retry-delay 2 --max-time 0 --output ./MOD00.A2021021.0835_1.PDS.bz2 https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/MOD00.A2021021.0835_1.PDS.bz2 % Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 291M 100 291M 0 0 63.4M 0 0:00:04 0:00:04 --:--:-- 83.6M
[bmurch@optics0 ~]$ rm ./MOD00.A2021021.0835_1.PDS.bz2
[bmurch@optics0 ~]$ curl -6 -b ~/.urs_cookies -c ~/.urs_cookies -L -n --retry 5 --retry-delay 2 --max-time 0 --output ./MOD00.A2021021.0835_1.PDS.bz2 https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/MOD00.A2021021.0835_1.PDS.bz2
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 291M 100 291M 0 0 51.6M 0 0:00:05 0:00:05 --:--:-- 65.1M
And yet again just now:
Worked:
[bmurch@optics0 l0]$ curl -b .urs_cookies_single -c .urs_cookies_single -L -n --interface 2607:fe50:0:6330::103 --retry 5 --retry-delay 2 --max-time 0 --output /cms_zfs/sat_data/modis/l0/2021/020/MOD00.P2021020.2315_1.PDS.bz2 https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/MOD00.P2021020.2315_1.PDS.bz2
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 70.8M 100 70.8M 0 0 22.9M 0 0:00:03 0:00:03 --:--:-- 54.5M
[bmurch@optics0 l0]$ file /cms_zfs/sat_data/modis/l0/2021/021/MOD00.P2021021.0010_1.PDS.bz2
Failed:
[bmurch@optics0 l0]$ curl -b .urs_cookies_single -c .urs_cookies_single -L -n --interface 2607:fe50:0:6330::103 --retry 5 --retry-delay 2 --max-time 0 --output /cms_zfs/sat_data/modis/l0/2021/021/MOD00.P2021021.0010_1.PDS.bz2 https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/MOD00.P2021021.0010_1.PDS.bz2
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
110 665 110 665 0 0 446 0 0:00:01 0:00:01 --:--:-- 1319
Warning: Transient problem: HTTP error Will retry in 2 seconds. 5 retries
Warning: left.
Throwing away 665 bytes
Then I up arrowed and tried again WORKED!:
[bmurch@optics0 l0]$ curl -b .urs_cookies_single -c .urs_cookies_single -L -n --interface 2607:fe50:0:6330::103 --retry 5 --retry-delay 2 --max-time 0 --output /cms_zfs/sat_data/modis/l0/2021/021/MOD00.P2021021.0010_1.PDS.bz2 https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/MOD00.P2021021.0010_1.PDS.bz2
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 279M 100 279M 0 0 57.8M 0 0:00:04 0:00:04 --:--:-- 80.2M
Worked:
[bmurch@optics0 l0]$ curl -b .urs_cookies_single -c .urs_cookies_single -L -n --interface 2607:fe50:0:6330::103 --retry 5 --retry-delay 2 --max-time 0 --output /cms_zfs/sat_data/modis/l0/2021/020/MOD00.P2021020.2315_1.PDS.bz2 https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/MOD00.P2021020.2315_1.PDS.bz2
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 70.8M 100 70.8M 0 0 22.9M 0 0:00:03 0:00:03 --:--:-- 54.5M
[bmurch@optics0 l0]$ file /cms_zfs/sat_data/modis/l0/2021/021/MOD00.P2021021.0010_1.PDS.bz2
Failed:
[bmurch@optics0 l0]$ curl -b .urs_cookies_single -c .urs_cookies_single -L -n --interface 2607:fe50:0:6330::103 --retry 5 --retry-delay 2 --max-time 0 --output /cms_zfs/sat_data/modis/l0/2021/021/MOD00.P2021021.0010_1.PDS.bz2 https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/MOD00.P2021021.0010_1.PDS.bz2
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
110 665 110 665 0 0 446 0 0:00:01 0:00:01 --:--:-- 1319
Warning: Transient problem: HTTP error Will retry in 2 seconds. 5 retries
Warning: left.
Throwing away 665 bytes
Then I up arrowed and tried again WORKED!:
[bmurch@optics0 l0]$ curl -b .urs_cookies_single -c .urs_cookies_single -L -n --interface 2607:fe50:0:6330::103 --retry 5 --retry-delay 2 --max-time 0 --output /cms_zfs/sat_data/modis/l0/2021/021/MOD00.P2021021.0010_1.PDS.bz2 https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/MOD00.P2021021.0010_1.PDS.bz2
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 279M 100 279M 0 0 57.8M 0 0:00:04 0:00:04 --:--:-- 80.2M
Same problem with OLCI data:
[bmurch@optics0 seadas_l1a_geo_extract_rt]$ file /cms_zfs/sat_data/olci/l1b/2021/020/S3* | grep HTML
/cms_zfs/sat_data/olci/l1b/2021/020/S3A_OL_1_ERR____20210120T070347_20210120T074738_20210121T104835_2631_067_277______LN1_O_NT_002.zip: HTML document text
/cms_zfs/sat_data/olci/l1b/2021/020/S3B_OL_1_ERR____20210120T062420_20210120T070810_20210121T104400_2630_048_134______LN1_O_NT_002.zip: HTML document text
/cms_zfs/sat_data/olci/l1b/2021/020/S3B_OL_1_ERR____20210120T080519_20210120T084909_20210121T122415_2630_048_135______LN1_O_NT_002.zip: HTML document text
/cms_zfs/sat_data/olci/l1b/2021/020/S3B_OL_1_ERR____20210120T094618_20210120T103008_20210121T140610_2630_048_136______LN1_O_NT_002.zip: HTML document text
/cms_zfs/sat_data/olci/l1b/2021/020/S3B_OL_1_ERR____20210120T112717_20210120T121107_20210121T154357_2630_048_137______LN1_O_NT_002.zip: HTML document text
[bmurch@optics0 seadas_l1a_geo_extract_rt]$ ll /cms_zfs/sat_data/olci/l1b/2021/020/S3A_OL_1_ERR____20210120T070347_20210120T074738_20210121T104835_2631_067_277______LN1_O_NT_002.zip
-rw-rw-r-- 1 bmurch cms_optics 665 Jan 21 12:35 /cms_zfs/sat_data/olci/l1b/2021/020/S3A_OL_1_ERR____20210120T070347_20210120T074738_20210121T104835_2631_067_277______LN1_O_NT_002.zip
[bmurch@optics0 l0]$ rm .urs_cookies_single; curl -b .urs_cookies_single -c .urs_cookies_single -L -n --interface 2607:fe50:0:6330::103 --retry 5 --retry-delay 2 --max-time 0 --output /cms_zfs/sat_data/olci/l1b/2021/020/S3B_OL_1_ERR____20210120T062420_20210120T070810_20210121T104400_2630_048_134______LN1_O_NT_002.zip https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/S3B_OL_1_ERR____20210120T062420_20210120T070810_20210121T104400_2630_048_134______LN1_O_NT_002.zip
rm: cannot remove `.urs_cookies_single': No such file or directory
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
110 665 110 665 0 0 738 0 --:--:-- --:--:-- --:--:-- 2309
Warning: Transient problem: HTTP error Will retry in 2 seconds. 5 retries
Warning: left.
^C
And then this one worked when I up arroed and hit enter:
[bmurch@optics0 l0]$ rm .urs_cookies_single; curl -b .urs_cookies_single -c .urs_cookies_single -L -n --interface 2607:fe50:0:6330::104 --retry 5 --retry-delay 2 --max-time 0 --output /cms_zfs/sat_data/olci/l1b/2021/020/S3B_OL_1_ERR____20210120T062420_20210120T070810_20210121T104400_2630_048_134______LN1_O_NT_002.zip https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/S3B_OL_1_ERR____20210120T062420_20210120T070810_20210121T104400_2630_048_134______LN1_O_NT_002.zip
rm: cannot remove `.urs_cookies_single': No such file or directory
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 681M 100 681M 0 0 73.5M 0 0:00:09 0:00:09 --:--:-- 94.8M
[bmurch@optics0 seadas_l1a_geo_extract_rt]$ file /cms_zfs/sat_data/olci/l1b/2021/020/S3* | grep HTML
/cms_zfs/sat_data/olci/l1b/2021/020/S3A_OL_1_ERR____20210120T070347_20210120T074738_20210121T104835_2631_067_277______LN1_O_NT_002.zip: HTML document text
/cms_zfs/sat_data/olci/l1b/2021/020/S3B_OL_1_ERR____20210120T062420_20210120T070810_20210121T104400_2630_048_134______LN1_O_NT_002.zip: HTML document text
/cms_zfs/sat_data/olci/l1b/2021/020/S3B_OL_1_ERR____20210120T080519_20210120T084909_20210121T122415_2630_048_135______LN1_O_NT_002.zip: HTML document text
/cms_zfs/sat_data/olci/l1b/2021/020/S3B_OL_1_ERR____20210120T094618_20210120T103008_20210121T140610_2630_048_136______LN1_O_NT_002.zip: HTML document text
/cms_zfs/sat_data/olci/l1b/2021/020/S3B_OL_1_ERR____20210120T112717_20210120T121107_20210121T154357_2630_048_137______LN1_O_NT_002.zip: HTML document text
[bmurch@optics0 seadas_l1a_geo_extract_rt]$ ll /cms_zfs/sat_data/olci/l1b/2021/020/S3A_OL_1_ERR____20210120T070347_20210120T074738_20210121T104835_2631_067_277______LN1_O_NT_002.zip
-rw-rw-r-- 1 bmurch cms_optics 665 Jan 21 12:35 /cms_zfs/sat_data/olci/l1b/2021/020/S3A_OL_1_ERR____20210120T070347_20210120T074738_20210121T104835_2631_067_277______LN1_O_NT_002.zip
[bmurch@optics0 l0]$ rm .urs_cookies_single; curl -b .urs_cookies_single -c .urs_cookies_single -L -n --interface 2607:fe50:0:6330::103 --retry 5 --retry-delay 2 --max-time 0 --output /cms_zfs/sat_data/olci/l1b/2021/020/S3B_OL_1_ERR____20210120T062420_20210120T070810_20210121T104400_2630_048_134______LN1_O_NT_002.zip https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/S3B_OL_1_ERR____20210120T062420_20210120T070810_20210121T104400_2630_048_134______LN1_O_NT_002.zip
rm: cannot remove `.urs_cookies_single': No such file or directory
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
110 665 110 665 0 0 738 0 --:--:-- --:--:-- --:--:-- 2309
Warning: Transient problem: HTTP error Will retry in 2 seconds. 5 retries
Warning: left.
^C
And then this one worked when I up arroed and hit enter:
[bmurch@optics0 l0]$ rm .urs_cookies_single; curl -b .urs_cookies_single -c .urs_cookies_single -L -n --interface 2607:fe50:0:6330::104 --retry 5 --retry-delay 2 --max-time 0 --output /cms_zfs/sat_data/olci/l1b/2021/020/S3B_OL_1_ERR____20210120T062420_20210120T070810_20210121T104400_2630_048_134______LN1_O_NT_002.zip https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/S3B_OL_1_ERR____20210120T062420_20210120T070810_20210121T104400_2630_048_134______LN1_O_NT_002.zip
rm: cannot remove `.urs_cookies_single': No such file or directory
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 681M 100 681M 0 0 73.5M 0 0:00:09 0:00:09 --:--:-- 94.8M
Brock,
Sometimes hardware behaves badly...one of our backend machines was acting up. Our firewall was occasionally (and in a kinda sticky manner) sending your traffic to that machine. The offending machine has been taken out back and shot....well removed from service until it's memory issues can be resolved.
Sean
Sometimes hardware behaves badly...one of our backend machines was acting up. Our firewall was occasionally (and in a kinda sticky manner) sending your traffic to that machine. The offending machine has been taken out back and shot....well removed from service until it's memory issues can be resolved.
Sean
Sean,
I understand hardware issues well.
I'm sending you a bunch of Log reports on another post as they are failing during processing.
Thanks,
Brock
I understand hardware issues well.
I'm sending you a bunch of Log reports on another post as they are failing during processing.
Thanks,
Brock
Dear Sir,
when I am running ::: modis_GEO.py filename.nc, I am getting these errors please provide a solution as soon as possible. I am a research scholar it will use for my research.
Traceback (most recent call last):
File "/home/umamahesh/ocssw/scripts/modis_GEO.py", line 7, in <module>
import modules.anc_utils as anc_utils
File "/home/umamahesh/ocssw/scripts/modules/anc_utils.py", line 9, in <module>
import modules.ProcUtils as ProcUtils
File "/home/umamahesh/ocssw/scripts/modules/ProcUtils.py", line 16, in <module>
import requests
ImportError: No module named requests
when I am running ::: modis_GEO.py filename.nc, I am getting these errors please provide a solution as soon as possible. I am a research scholar it will use for my research.
Traceback (most recent call last):
File "/home/umamahesh/ocssw/scripts/modis_GEO.py", line 7, in <module>
import modules.anc_utils as anc_utils
File "/home/umamahesh/ocssw/scripts/modules/anc_utils.py", line 9, in <module>
import modules.ProcUtils as ProcUtils
File "/home/umamahesh/ocssw/scripts/modules/ProcUtils.py", line 16, in <module>
import requests
ImportError: No module named requests
Your question is not related to this thread, which deals with problems obtaining ancillary data. I am starting a new thread called "OCSSW processing: ImportError: No module named requests from modis_GEO.py". Please read that post.
Topic Products and Algorithms / Satellite Data Access / problems with MODIS / VIIRS Data access some anc data d/ls fine
Powered by mwForum 2.29.7 © 1999-2015 Markus Wichitill