Not logged inOcean Color Forum

The forum is locked.

The Ocean Color Forum has transitioned over to the Earthdata Forum (https://forum.earthdata.nasa.gov/). The information existing below will be retained for historical reference. Please sign into the Earthdata Forum for active user support.

Up Topic Products and Algorithms / Satellite Data Access / problems with MODIS / VIIRS Data access some anc data d/ls fine
- By oo_processing Date 2021-01-21 11:15
Lo all,

Do you have problems going on with various data (MODIS / VIIRS) Data access?

I can't seem to get files across several installations home/saint petersburg/tampa but I can get some ancillary data like N2021020_SST_OIV2AV_24h.nc?

I can get this:
$ curl -b ~/.urs_cookies -c ~/.urs_cookies -L -n --retry 5 --retry-delay 2 --max-time 0 --output ./N2021020_SST_OIV2AV_24h.nc https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/N2021020_SST_OIV2AV_24h.nc
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100 1619k  100 1619k    0     0   708k      0  0:00:02  0:00:02 --:--:-- 2890k

But not this:
$ curl -b .urs_cookies_single -c .urs_cookies_single -L -n --output ./PM1EPHND_NRT.A2021020.1555.06 https://oceandata.sci.gsfc.nasa.gov/cgi/PM1EPHND_NRT.A2021020.1555.06
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   665  100   665    0     0    960      0 --:--:-- --:--:-- --:--:--   959

Or any MODIS/VIIRS

$ curl -b ~/.urs_cookies -c ~/.urs_cookies -L -n --retry 5 --retry-delay 2 --max-time 0 --output ./MOD00.P2021021.0725_1.PDS.bz2 https://oceandata.sci.gsfc.nasa.gov/cgi/MOD00.P2021021.0725_1.PDS.bz2
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   665  100   665    0     0   1205      0 --:--:-- --:--:-- --:--:--  1206
$ curl -b ~/.urs_cookies -c ~/.urs_cookies -L -n --retry 5 --retry-delay 2 --max-time 0 --output ./MOD00.A2021020.1850_1.PDS.bz2 https://oceandata.sci.gsfc.nasa.gov/cgi/MOD00.A2021020.1850_1.PDS.bz2
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   665  100   665    0     0   1203      0 --:--:-- --:--:-- --:--:--  1202

$ curl -b .urs_cookies_single -c .urs_cookies_single -L -n --output ./V2021020045400.L1A_SNPP.nc https://oceandata.sci.gsfc.nasa.gov/cgi/V2021020045400.L1A_SNPP.nc
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   665  100   665    0     0   1180      0 --:--:-- --:--:-- --:--:--  1181

$ file *
MOD00.A2021020.1850_1.PDS.bz2: HTML document, ASCII text, with very long lines
MOD00.P2021021.0725_1.PDS.bz2: HTML document, ASCII text, with very long lines
N2021020_SST_OIV2AV_24h.nc:    bzip2 compressed data, block size = 100k
PM1EPHND_NRT.A2021020.1555.06: HTML document, ASCII text, with very long lines
V2021020045400.L1A_SNPP.nc: HTML document, ASCII text, with very long lines
$ bunzip2 N2021020_SST_OIV2AV_24h.nc
bunzip2: Can't guess original name for N2021020_SST_OIV2AV_24h.nc -- using N2021020_SST_OIV2AV_24h.nc.out
$ mv N2021020_SST_OIV2AV_24h.nc.out N2021020_SST_OIV2AV_24h.nc
$ file N2021020_SST_OIV2AV_24h.nc
N2021020_SST_OIV2AV_24h.nc: Hierarchical Data Format (version 5) data
- By jgwilding Date 2021-01-21 11:27
The URLs on your failed examples seem to be missing the /getfile/ element of the URL.  The first successful case has it .../cgi/getfile/name-of-file but the others have .../cgi/name-of-file

john
- By oo_processing Date 2021-01-21 11:48 Edited 2021-01-21 11:52
John,

I think it was a copy paste thing again but here not on the command line again:

From home worked twice:
[bmurch@dell8 test]$ curl -b ./.urs_cookies_get_viirs -c ./.urs_cookies_get_viirs -L -n --retry 5 --retry-delay 2 --max-time 0 --output ./V2020153180600.L1A_SNPP.nc https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/V2020153180600.L1A_SNPP.nc
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100  451M  100  451M    0     0  24.6M      0  0:00:18  0:00:18 --:--:-- 29.6M
[bmurch@dell8 test]$ curl -b ~/.urs_cookies -c ~/.urs_cookies -L -n --retry 5 --retry-delay 2 --max-time 0 --output ./MOD00.P2021021.0725_1.PDS.bz2 https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/MOD00.P2021021.0725_1.PDS.bz2
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100 71.0M  100 71.0M    0     0  15.0M      0  0:00:04  0:00:04 --:--:-- 25.2M

But not this one:
[bmurch@dell8 test]$ curl -b ~/.urs_cookies -c ~/.urs_cookies -L -n --retry 5 --retry-delay 2 --max-time 0 --output ./MOD00.A2021021.0835_1.PDS.bz2 https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/MOD00.A2021021.0835_1.PDS.bz2
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   665  100   665    0     0     16      0  0:00:41  0:00:41 --:--:--   151
Warning: Transient problem: HTTP error Will retry in 2 seconds. 5 retries
Warning: left.
Throwing away 665 bytes
....

Same files from work (131.247.138.46)
curl -b ~/.urs_cookies -c ~/.urs_cookies -L -n --retry 5 --retry-delay 2 --max-time 0 --output ./MOD00.P2021021.0725_1.PDS.bz2 https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/MOD00.P2021021.0725_1.PDS.bz2
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   665  100   665    0     0     16      0  0:00:41  0:00:41 --:--:--   208
Warning: Transient problem: HTTP error Will retry in 2 seconds. 5 retries
Warning: left.
....

curl -b ./.urs_cookies_get_viirs -c ./.urs_cookies_get_viirs -L -n --retry 5 --retry-delay 2 --max-time 0 --output ./V2020153180600.L1A_SNPP.nc https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/V2020153180600.L1A_SNPP.nc
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   665  100   665    0     0     16      0  0:00:41  0:00:41 --:--:--   156
Warning: Transient problem: HTTP error Will retry in 2 seconds. 5 retries
Warning: left.
Throwing away 665 bytes
....

From work 2607:fe50:0:6330::103:
These are coming down now as 665 (and 0 length until they arrive as 665 size html)
-rw-rw-r-- 1 bmurch cms_optics       665 Jan 21 11:38 /cms_zfs/sat_data/modis/l0/2021/021/MOD00.A2021021.0835_1.PDS.bz2
-rw-rw-r-- 1 bmurch cms_optics       665 Jan 21 11:34 /cms_zfs/sat_data/modis/l0/2021/021/MOD00.A2021021.0850_1.PDS.bz2
-rw-rw-r-- 1 bmurch cms_optics         0 Jan 21 11:42 /cms_zfs/sat_data/modis/l0/2021/021/MOD00.A2021021.0930_1.PDS.bz2
-rw-rw-r-- 1 bmurch cms_optics       665 Jan 20 21:00 /cms_zfs/sat_data/modis/l0/2021/021/MOD00.P2021021.0010_1.PDS.bz2
- By jgwilding Date 2021-01-21 12:04
Can you view the HTML in a browser to see what it is asking for, such as maybe an Earthdata login prompt?  Verify .netrc values?  Remove the ./.urs_cookies_get_* files to force a new one?

I can't replicate the problem, so the getfile service is functioning.  You're saying it is sporadically failing on two different machines, or always working on one, failing on another?

john
- By oo_processing Date 2021-01-21 12:11 Edited 2021-01-21 12:14
It might be IPv6 as well:
Here I try with -4 and it works
[bmurch@optics0 ~]$ curl -4 -b ~/.urs_cookies -c ~/.urs_cookies -L -n --retry 5 --retry-delay 2 --max-time 0 --output ./MOD00.A2021021.0835_1.PDS.bz2 https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/MOD00.A2021021.0835_1.PDS.bz2
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  291M  100  291M    0     0  61.4M      0  0:00:04  0:00:04 --:--:-- 87.3M

[bmurch@optics0 ~]$ rm ./MOD00.A2021021.0835_1.PDS.bz2

Here I try with -6 and it fails:
[bmurch@optics0 ~]$ curl -6 -b ~/.urs_cookies -c ~/.urs_cookies -L -n --retry 5 --retry-delay 2 --max-time 0 --output ./MOD00.A2021021.0835_1.PDS.bz2 https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/MOD00.A2021021.0835_1.PDS.bz2
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
110   665  110   665    0     0    901      0 --:--:-- --:--:-- --:--:--  4650
Warning: Transient problem: HTTP error Will retry in 2 seconds. 5 retries
Warning: left.
^C

[bmurch@optics0 ~]$ rm ./MOD00.A2021021.0835_1.PDS.bz2

Here I try again with -4 and it works again:
[bmurch@optics0 ~]$ curl -4 -b ~/.urs_cookies -c ~/.urs_cookies -L -n --retry 5 --retry-delay 2 --max-time 0 --output ./MOD00.A2021021.0835_1.PDS.bz2 https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/MOD00.A2021021.0835_1.PDS.bz2
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  291M  100  291M    0     0  59.7M      0  0:00:04  0:00:04 --:--:-- 80.8M

[bmurch@optics0 ~]$ rm ./MOD00.A2021021.0835_1.PDS.bz2

Here I try again with -6 and it fails:
[bmurch@optics0 ~]$ curl -6 -b ~/.urs_cookies -c ~/.urs_cookies -L -n --retry 5 --retry-delay 2 --max-time 0 --output ./MOD00.A2021021.0835_1.PDS.bz2 https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/MOD00.A2021021.0835_1.PDS.bz2
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
110   665  110   665    0     0    498      0  0:00:01  0:00:01 --:--:--  1324
Warning: Transient problem: HTTP error Will retry in 2 seconds. 5 retries
Warning: left.
Throwing away 665 bytes
  0   665    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0^C

And fails with the interface specified as well:
$ curl --interface 2607:fe50:0:6330::102 -b ~/.urs_cookies -c ~/.urs_cookies -L -n --retry 5 --retry-delay 2 --max-time 0 --output ./MOD00.A2021021.0835_1.PDS.bz2 https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/MOD00.A2021021.0835_1.PDS.bz2
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
110   665  110   665    0     0    799      0 --:--:-- --:--:-- --:--:--  2309
Warning: Transient problem: HTTP error Will retry in 2 seconds. 5 retries
Warning: left.
^C
- By oo_processing Date 2021-01-21 12:22
I was having issues on all machines with sporadic downloads. I just tried again, and it worked:

[bmurch@optics0 ~]$ curl -4 -b ~/.urs_cookies -c ~/.urs_cookies -L -n --retry 5 --retry-delay 2 --max-time 0 --output ./MOD00.A2021021.0835_1.PDS.bz2 https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/MOD00.A2021021.0835_1.PDS.bz2  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  291M  100  291M    0     0  63.4M      0  0:00:04  0:00:04 --:--:-- 83.6M
[bmurch@optics0 ~]$ rm ./MOD00.A2021021.0835_1.PDS.bz2
[bmurch@optics0 ~]$ curl -6 -b ~/.urs_cookies -c ~/.urs_cookies -L -n --retry 5 --retry-delay 2 --max-time 0 --output ./MOD00.A2021021.0835_1.PDS.bz2 https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/MOD00.A2021021.0835_1.PDS.bz2
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  291M  100  291M    0     0  51.6M      0  0:00:05  0:00:05 --:--:-- 65.1M
- By oo_processing Date 2021-01-21 12:39
And yet again just now:

Worked:
[bmurch@optics0 l0]$ curl -b .urs_cookies_single -c .urs_cookies_single -L -n --interface 2607:fe50:0:6330::103 --retry 5 --retry-delay 2 --max-time 0  --output /cms_zfs/sat_data/modis/l0/2021/020/MOD00.P2021020.2315_1.PDS.bz2 https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/MOD00.P2021020.2315_1.PDS.bz2
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 70.8M 100 70.8M    0     0  22.9M      0  0:00:03  0:00:03 --:--:-- 54.5M
[bmurch@optics0 l0]$ file /cms_zfs/sat_data/modis/l0/2021/021/MOD00.P2021021.0010_1.PDS.bz2

Failed:
[bmurch@optics0 l0]$ curl -b .urs_cookies_single -c .urs_cookies_single -L -n --interface 2607:fe50:0:6330::103 --retry 5 --retry-delay 2 --max-time 0  --output /cms_zfs/sat_data/modis/l0/2021/021/MOD00.P2021021.0010_1.PDS.bz2 https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/MOD00.P2021021.0010_1.PDS.bz2
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
110   665  110   665    0     0    446      0  0:00:01  0:00:01 --:--:--  1319
Warning: Transient problem: HTTP error Will retry in 2 seconds. 5 retries
Warning: left.
Throwing away 665 bytes

Then I up arrowed and tried again WORKED!:
[bmurch@optics0 l0]$ curl -b .urs_cookies_single -c .urs_cookies_single -L -n --interface 2607:fe50:0:6330::103 --retry 5 --retry-delay 2 --max-time 0  --output /cms_zfs/sat_data/modis/l0/2021/021/MOD00.P2021021.0010_1.PDS.bz2 https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/MOD00.P2021021.0010_1.PDS.bz2
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  279M  100  279M    0     0  57.8M      0  0:00:04  0:00:04 --:--:-- 80.2M
- By oo_processing Date 2021-01-21 12:48
Same problem with OLCI data:
[bmurch@optics0 seadas_l1a_geo_extract_rt]$ file /cms_zfs/sat_data/olci/l1b/2021/020/S3* | grep HTML
/cms_zfs/sat_data/olci/l1b/2021/020/S3A_OL_1_ERR____20210120T070347_20210120T074738_20210121T104835_2631_067_277______LN1_O_NT_002.zip: HTML document text
/cms_zfs/sat_data/olci/l1b/2021/020/S3B_OL_1_ERR____20210120T062420_20210120T070810_20210121T104400_2630_048_134______LN1_O_NT_002.zip: HTML document text
/cms_zfs/sat_data/olci/l1b/2021/020/S3B_OL_1_ERR____20210120T080519_20210120T084909_20210121T122415_2630_048_135______LN1_O_NT_002.zip: HTML document text
/cms_zfs/sat_data/olci/l1b/2021/020/S3B_OL_1_ERR____20210120T094618_20210120T103008_20210121T140610_2630_048_136______LN1_O_NT_002.zip: HTML document text
/cms_zfs/sat_data/olci/l1b/2021/020/S3B_OL_1_ERR____20210120T112717_20210120T121107_20210121T154357_2630_048_137______LN1_O_NT_002.zip: HTML document text
[bmurch@optics0 seadas_l1a_geo_extract_rt]$ ll /cms_zfs/sat_data/olci/l1b/2021/020/S3A_OL_1_ERR____20210120T070347_20210120T074738_20210121T104835_2631_067_277______LN1_O_NT_002.zip
-rw-rw-r-- 1 bmurch cms_optics 665 Jan 21 12:35 /cms_zfs/sat_data/olci/l1b/2021/020/S3A_OL_1_ERR____20210120T070347_20210120T074738_20210121T104835_2631_067_277______LN1_O_NT_002.zip

[bmurch@optics0 l0]$ rm .urs_cookies_single;  curl -b .urs_cookies_single -c .urs_cookies_single -L -n --interface 2607:fe50:0:6330::103 --retry 5 --retry-delay 2 --max-time 0  --output /cms_zfs/sat_data/olci/l1b/2021/020/S3B_OL_1_ERR____20210120T062420_20210120T070810_20210121T104400_2630_048_134______LN1_O_NT_002.zip https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/S3B_OL_1_ERR____20210120T062420_20210120T070810_20210121T104400_2630_048_134______LN1_O_NT_002.zip
rm: cannot remove `.urs_cookies_single': No such file or directory
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
110   665  110   665    0     0    738      0 --:--:-- --:--:-- --:--:--  2309
Warning: Transient problem: HTTP error Will retry in 2 seconds. 5 retries
Warning: left.
^C
And then this one worked when I up arroed and hit enter:
[bmurch@optics0 l0]$ rm .urs_cookies_single;  curl -b .urs_cookies_single -c .urs_cookies_single -L -n --interface 2607:fe50:0:6330::104 --retry 5 --retry-delay 2 --max-time 0  --output /cms_zfs/sat_data/olci/l1b/2021/020/S3B_OL_1_ERR____20210120T062420_20210120T070810_20210121T104400_2630_048_134______LN1_O_NT_002.zip https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/S3B_OL_1_ERR____20210120T062420_20210120T070810_20210121T104400_2630_048_134______LN1_O_NT_002.zip
rm: cannot remove `.urs_cookies_single': No such file or directory
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  681M  100  681M    0     0  73.5M      0  0:00:09  0:00:09 --:--:-- 94.8M
- By seanbailey Date 2021-01-21 13:57
Brock,

Sometimes hardware behaves badly...one of our backend machines was acting up.  Our firewall was occasionally (and in a kinda sticky manner) sending your traffic to that machine.  The offending machine has been taken out back and shot....well removed from service until it's memory issues can be resolved. 

Sean
- By oo_processing Date 2021-01-21 14:06
Sean,

I understand hardware issues well.

I'm sending you a bunch of Log reports on another post as they are failing during processing.

Thanks,
Brock
- By maheshgeotech38 Date 2021-01-30 03:52
Dear Sir,
             when I am running  ::: modis_GEO.py filename.nc, I am getting these errors please provide a solution as soon as possible. I am a research scholar it will use for my research.

Traceback (most recent call last):
  File "/home/umamahesh/ocssw/scripts/modis_GEO.py", line 7, in <module>
    import modules.anc_utils as anc_utils
  File "/home/umamahesh/ocssw/scripts/modules/anc_utils.py", line 9, in <module>
    import modules.ProcUtils as ProcUtils
  File "/home/umamahesh/ocssw/scripts/modules/ProcUtils.py", line 16, in <module>
    import requests
ImportError: No module named requests
- By gnwiii Date 2021-01-30 10:02
Your question is not related to this thread, which deals with problems obtaining ancillary data.   I am starting a new thread called "OCSSW processing: ImportError: No module named requests from modis_GEO.py".  Please read that post.
Up Topic Products and Algorithms / Satellite Data Access / problems with MODIS / VIIRS Data access some anc data d/ls fine

Powered by mwForum 2.29.7 © 1999-2015 Markus Wichitill