Not logged inOcean Color Forum

The forum is locked.

The Ocean Color Forum has transitioned over to the Earthdata Forum (https://forum.earthdata.nasa.gov/). The information existing below will be retained for historical reference. Please sign into the Earthdata Forum for active user support.

Up Topic Products and Algorithms / Satellite Data Access / wget only downloading partial file
- By ermill99 Date 2020-02-17 04:15
I'm using WGET to download L2 files to my server.  The files are showing up but seem to be capped at 13 kb.  Any idea why?

WGET Command
wget -P ~/www/saltwx.com/public_html/MODIS/sst/ --user=my-user--password='my-pw' --auth-no-challenge=on "https://oceandata.sci.gsfc.nasa.gov/ob/getfile/A2020046181000.L2_LAC.S3470.nc.Southeast.sst.png"

Result

--2020-02-17 02:51:01--  https://oceandata.sci.gsfc.nasa.gov/ob/getfile/A2020046181000.L2_LAC.S3470.nc.Southeast.sst.png
Resolving oceandata.sci.gsfc.nasa.gov (oceandata.sci.gsfc.nasa.gov)... 169.154.128.84, 2001:4d0:2418:128::84
Connecting to oceandata.sci.gsfc.nasa.gov (oceandata.sci.gsfc.nasa.gov)|169.154.128.84|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://urs.earthdata.nasa.gov/oauth/authorize?redirect_uri=https%3A%2F%2Foceandata.sci.gsfc.nasa.gov%2Fob%2Fgetfile%2Frestrict&response_type=code&client_id=Z0u-MdLNypXBjiDREZ3roA [following]
--2020-02-17 02:51:01--  https://urs.earthdata.nasa.gov/oauth/authorize?redirect_uri=https%3A%2F%2Foceandata.sci.gsfc.nasa.gov%2Fob%2Fgetfile%2Frestrict&response_type=code&client_id=Z0u-MdLNypXBjiDREZ3roA
Resolving urs.earthdata.nasa.gov (urs.earthdata.nasa.gov)... 198.118.243.33, 2001:4d0:241a:4081::89
Connecting to urs.earthdata.nasa.gov (urs.earthdata.nasa.gov)|198.118.243.33|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: '/home/my_key/www/saltwx.com/public_html/MODIS/sst/A2020046181000.L2_LAC.S3470.nc.Southeast.sst.png'

     0K .......... ..                                           381K=0.03s

2020-02-17 02:51:01 (381 KB/s) - '/home/my_key/www/saltwx.com/public_html/MODIS/sst/A2020046181000.L2_LAC.S3470.nc.Southeast.sst.png'
saved [12671] <----This appears to be the total file size downloaded.
- By jgwilding Date 2020-02-17 14:45
This usually indicates that an HTML file was downloaded.  If you're on a Linux platform, you can try the 'file' command on the saved file.  If it is HTML, you should be able to view it with a text viewer or in a browser.  A lot of times this ends up being the Earthdata login page and might mean your login credentials are not being accepted.

john
- By gnwiii Date 2020-02-17 14:54
Rather than a partial download you may getting an html document you can view in a browser.   Try adding the wget --adjust-extension option.   You may need the --load-cookies ~/.urs_cookies --save-cookies ~/.urs_cookies --keep-session-cookies options.
- By ermill99 Date 2020-02-17 16:55 Edited 2020-02-17 17:11
So, like this?

wget -P ~/www/saltwx.com/public_html/MODIS/sst/ --adjust-extension --user=user--password='password' --auth-no-challenge=on "https://oceandata.sci.gsfc.nasa.gov/ob/getfile/A2020046181000.L2_LAC.S3470.nc.Southeast.sst.png"

OR

wget -P ~/www/saltwx.com/public_html/MODIS/sst/ --adjust-extension --load-cookies ~/.urs_cookies --save-cookies ~/.urs_cookies --keep-session-cookies "https://oceandata.sci.gsfc.nasa.gov/ob/getfile/A2020046181000.L2_LAC.S3470.nc.Southeast.sst.png"

Postscript: I able was to open the file and it is indeed the login page.  I'm beginning to feel that there is no way to simply download a PNG file with WGET.
- By ermill99 Date 2020-02-17 18:35
I've tried all suggestions and just cannot get the single file I would like, to actually download.   

Are there any other suggestions to how to get this file : A2020046181000.L2_LAC.S3470.nc.Southeast.sst.png to download non-interactively?

All that I am getting is the login page html downloaded to my server.
- By gnwiii Date 2020-02-17 18:44
Wget (as well as wget2) also fails for me, but downloads succeed using a browser or the following curl incantation (from Download Methods):

curl -O -b ~/.urs_cookies -c ~/.urs_cookies -L -n https://oceandata.sci.gsfc.nasa.gov/ob/getfile/A2020046181000.L2_LAC.S3470.nc.Southeast.sst.png
- By ermill99 Date 2020-02-17 19:02
here are the results of both methods and there appears to be no difference in outcome.  Still only getting the HTML file

1
Command
wget -P ~/www/saltwx.com/public_html/MODIS/sst/ --user=user--password='PW' --auth-no-challenge=on "https://oceandata.sci.gsfc.nasa.gov/ob/getfile/A2020046181000.L2_LAC.S3470.nc.Southeast.sst.png"

Result
--2020-02-17 18:50:01--  https://oceandata.sci.gsfc.nasa.gov/ob/getfile/A2020046181000.L2_LAC.S3470.nc.Southeast.sst.png
Resolving oceandata.sci.gsfc.nasa.gov (oceandata.sci.gsfc.nasa.gov)... 169.154.128.84, 2001:4d0:2418:128::84
Connecting to oceandata.sci.gsfc.nasa.gov (oceandata.sci.gsfc.nasa.gov)|169.154.128.84|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://urs.earthdata.nasa.gov/oauth/authorize?redirect_uri=https%3A%2F%2Foceandata.sci.gsfc.nasa.gov%2Fob%2Fgetfile%2Frestrict&response_type=code&client_id=Z0u-MdLNypXBjiDREZ3roA&required_scope=study_area,country [following]
--2020-02-17 18:50:02--  https://urs.earthdata.nasa.gov/oauth/authorize?redirect_uri=https%3A%2F%2Foceandata.sci.gsfc.nasa.gov%2Fob%2Fgetfile%2Frestrict&response_type=code&client_id=Z0u-MdLNypXBjiDREZ3roA&required_scope=study_area,country
Resolving urs.earthdata.nasa.gov (urs.earthdata.nasa.gov)... 198.118.243.33, 2001:4d0:241a:4081::89
Connecting to urs.earthdata.nasa.gov (urs.earthdata.nasa.gov)|198.118.243.33|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: '/home/u250-zd3apyn5m53e/www/saltwx.com/public_html/MODIS/sst/A2020046181000.L2_LAC.S3470.nc.Southeast.sst.png.2'

     0K .......... ..                                          41.1M=0s

2020-02-17 18:50:02 (41.1 MB/s) - '/home/u250-zd3apyn5m53e/www/saltwx.com/public_html/MODIS/sst/A2020046181000.L2_LAC.S3470.nc.Southeast.sst.png.2' saved [12859]

2
Command
wget -P ~/www/saltwx.com/public_html/MODIS/sst/ --user=user--password='pw' --load-cookies ~/.urs_cookies --save-cookies ~/.urs_cookies --keep-session-cookies --auth-no-challenge=on "https://oceandata.sci.gsfc.nasa.gov/ob/getfile/A2020046181000.L2_LAC.S3470.nc.Southeast.sst.png"

Result
--2020-02-17 18:40:01--  https://oceandata.sci.gsfc.nasa.gov/ob/getfile/A2020046181000.L2_LAC.S3470.nc.Southeast.sst.png
Resolving oceandata.sci.gsfc.nasa.gov (oceandata.sci.gsfc.nasa.gov)... 169.154.128.84, 2001:4d0:2418:128::84
Connecting to oceandata.sci.gsfc.nasa.gov (oceandata.sci.gsfc.nasa.gov)|169.154.128.84|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://urs.earthdata.nasa.gov/oauth/authorize?redirect_uri=https%3A%2F%2Foceandata.sci.gsfc.nasa.gov%2Fob%2Fgetfile%2Frestrict&response_type=code&client_id=Z0u-MdLNypXBjiDREZ3roA&required_scope=study_area,country [following]
--2020-02-17 18:40:08--  https://urs.earthdata.nasa.gov/oauth/authorize?redirect_uri=https%3A%2F%2Foceandata.sci.gsfc.nasa.gov%2Fob%2Fgetfile%2Frestrict&response_type=code&client_id=Z0u-MdLNypXBjiDREZ3roA&required_scope=study_area,country
Resolving urs.earthdata.nasa.gov (urs.earthdata.nasa.gov)... 198.118.243.33, 2001:4d0:241a:4081::89
Connecting to urs.earthdata.nasa.gov (urs.earthdata.nasa.gov)|198.118.243.33|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: '/home/u250-zd3apyn5m53e/www/saltwx.com/public_html/MODIS/sst/A2020046181000.L2_LAC.S3470.nc.Southeast.sst.png.1'

     0K .......... ..                                          56.3M=0s

2020-02-17 18:40:08 (56.3 MB/s) - '/home/u250-zd3apyn5m53e/www/saltwx.com/public_html/MODIS/sst/A2020046181000.L2_LAC.S3470.nc.Southeast.sst.png.1' saved [12859]
- By ermill99 Date 2020-02-17 19:37 Edited 2020-02-17 19:41
OK, the cURL seemed to download but I think it is still only delivering the HTML page.

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0

100 12859    0 12859    0     0  38157      0 --:--:-- --:--:-- --:--:-- 38157
- By ermill99 Date 2020-02-17 20:04
Confirmed that each method, WGET and cURL, result in a 12.859 KB file download. 

This authentication process is ridiculous.
- By seanbailey Date 2020-02-17 20:32
The cURL method requires a .netrc file (wget can also use this file, but you will need the --auth-no-challenge=on option) that contains your login Earthdata credentials.
Verify you have one, and it is correct, or generate one ala:

echo "machine urs.earthdata.nasa.gov login USERNAME password PASSWD" > ~/.netrc
chmod  0600 ~/.netrc


https://oceancolor.gsfc.nasa.gov/data/download_methods/ describes the process
Sean
- By ermill99 Date 2020-02-17 21:34
Thanks Sean.  Just to clarify I'm doing the following:

1) saving my .netrc file to the my server where I am writing the file to 

2) Running the wget command

wget -P ~/www/saltwx.com/public_html/MODIS/sst/ --load-cookies ~/.urs_cookies --save-cookies ~/.urs_cookies --keep-session-cookies --auth-no-challenge=on "https://oceandata.sci.gsfc.nasa.gov/ob/getfile/A2020046181000.L2_LAC.S3470.nc.Southeast.sst.png"

OR

the cURL command
curl -O -b ~/.urs_cookies -c ~/.urs_cookies -L -n https://oceandata.sci.gsfc.nasa.gov/ob/getfile/A2020046181000.L2_LAC.S3470.nc.Southeast.sst.png
- By seanbailey Date 2020-02-17 23:01
Yes, as long as the .netrc file is in the home directory for the account on the server that will be running the wget/curl commands, it should work.
If you still have issues getting it to work,  run the commands with verbose -v or debugging on (-d, --debug for wget , --trace) to see what's going on.

Sean
- By gnwiii Date 2020-02-18 15:03
Some software won't use a "~/.netrc" if it has incorrect permissions, so good practice is "chmod 600 ~/.netrc".   If "~/.netrc" was created with an overly helpful editor it may contain Unicode look-alike characters in place of quotes, dashes, etc.  EarthData WIKI:  Troubleshooting Wget mentions a perl script to create a "~/.netrc".
- By ermill99 Date 2020-02-20 21:09
Do I need ~/.netrc or hmod 600 ~/.netrc in my curl command? 

I am still only getting small size files (27 b) to download to my folder.

I have my .netrc file in the destination folder plus the home directory. Heck, I think I have it every possible folder.  Still no complete file is downloading. 

I am using the following command

curl https://oceandata.sci.gsfc.nasa.gov/ob/getfile/A2020046181000.L2_LAC.S3470.nc.Southeast.sst.png -b ~/.urs_cookies -c ~/.urs_cookies -L -n -o ~/www/saltwx.com/public_html/MODIS/sst/A2020046181000.L2_LAC.S3470.nc.Southeast.sst.png
- By amscott Date 2020-02-20 21:34
wget -P ~/www/saltwx.com/public_html/MODIS/sst/ --load-cookies ~/.urs_cookies --save-cookies ~/.urs_cookies --keep-session-cookies --auth-no-challenge=on https://oceandata.sci.gsfc.nasa.gov/ob/getfile/A2020046181000.L2_LAC.S3470.nc.Southeast.sst.png

works fine. You need to remove the quotes around the URL.
- By amscott Date 2020-02-20 21:36 Edited 2020-02-20 21:43
Do I need ~/.netrc or hmod 600 ~/.netrc in my curl command? NO

With the curl command:
curl https://oceandata.sci.gsfc.nasa.gov/ob/getfile/A2020046181000.L2_LAC.S3470.nc.Southeast.sst.png -b ~/.urs_cookies -c ~/.urs_cookies -L -n -o ~/www/saltwx.com/public_html/MODIS/sst/A2020046181000.L2_LAC.S3470.nc.Southeast.sst.png

Create the directory first, then it will put the file there. --create-dirs should work, but it didn't for me.
- By ermill99 Date 2020-02-20 23:20
I keep getting Username/Password Authentication Failed.  when using Wget
- By gnwiii Date 2020-02-21 00:59
"~/.netrc" or "chmod 600 ~/.netrc are not curl options.   The "~" is a bash shorthand for the user's HOME directory.  Curl and wget both check for "~/.netrc".  Problems with "~/.netrc" file will not affect downloads with credentials provided on the command line.   Since the contents are sensitive, you should delete all the other copies.   Both wget and curl have options to provide more details of the rather complicated 9 step download process.  
Try setting verbose output and dumping the http headers to, e.g. /tmp/headers:

curl https://oceandata.sci.gsfc.nasa.gov/ob/getfile/A2020046181000.L2_LAC.S3470.nc.Southeast.sst.png -D /tmp/headers -v -b ~/.urs_cookies -c ~/.urs_cookies -L -n -o ~/www/saltwx.com/public_html/MODIS/sst/A2020046181000.L2_LAC.S3470.nc.Southeast.sst.png

For me, this gives a lot of detail and a successful download (using /tmp in place of the above output directory).  The sequence of headers was:
$ grep '^HTTP' /tmp/headers
HTTP/2 302
HTTP/1.1 302 Found
HTTP/2 302
HTTP/2 200


The output will have messages like:
* Couldn't find host oceandata.sci.gsfc.nasa.gov in the .netrc file; using defaults which you can ignore since the login is done on ura.earthdata.nasa.gov.

If you get something different there is a lot more information in the output, but also some sensitive information.

If you have the OCSSW processing software, you could try getanc.py.   If that works, you could adapt the second python script from  How to Access Data with Python.  This uses python3 and the requests library (generally available as a distro package for linux and may already be installed on your system). You will need to edit the script to provide your credentials and replace the URL with the one you want.  I used this example as the basis for a python script that gets credentials from "~/.netrc".   Unlike wget, it has never failed for me in tests on several platforms (Windows cygwin64, macOS, and 3 linux distros).
- By seanbailey Date 2020-02-21 18:18
did you try the suggestions I offered earlier to help diagnose the problem?
https://oceancolor.gsfc.nasa.gov/forum/oceancolor/topic_show.pl?pid=52248#pid52248
Up Topic Products and Algorithms / Satellite Data Access / wget only downloading partial file

Powered by mwForum 2.29.7 © 1999-2015 Markus Wichitill