Not logged inOcean Color Forum

The forum is locked.

The Ocean Color Forum has transitioned over to the Earthdata Forum (https://forum.earthdata.nasa.gov/). The information existing below will be retained for historical reference. Please sign into the Earthdata Forum for active user support.

Up Topic SeaDAS / SeaDAS - General Questions / l2bin error
- By Warner9392 Date 2017-05-17 16:54 Edited 2017-05-18 08:56
Greetings
I am trying to process a number of L2 oc (MODIS Aqua) files I ordered and downloaded.

I am running l2bin on individual wavelengths, starting with 207 original scenes (files).  The line of code that gets submitted via command line is...

dmwarner$ /Users/dmwarner/Documents/MODIS/scripts/l2bin_lake.sh MODIS Michigan Rrs_443 2016

I am not getting that 207 L3bin files.  And I am getting different numbers of L3 bin files depending on the wavelength I process.

I noticed that during the processing that there was the following result multiple times.  This happens for each wavelength, some more than others, but none of them provide me with the full 207 L3 binned scenes.  This is causing some problems for my processing chain and I would like to understand this behavior.

Working on A2016321182500.L2_LAC_OC.x.nc
Defined A2016321182500.L2_LAC_OC.x.nc
L2BIN 4.0.9 (Apr 27 2016 10:22:37)
INTERP parameter disabled.
Single HDF input
Averaging: standard
Resolution: 1
Max Qual Allowed: 2
prod_avg:
flagusemask: 33559043
required: 0
NetCDF4 input file
-W- /Users/seadas/ocssw/build/src/l2bin/dataday.c line 294: Insufficient valid latitudes found
Consider QC fail for file: A2016321182500.L2_LAC_OC.x.nc
...look at the file though, as I might be lying...
/Users/dmwarner/Documents/MODIS/scripts/l2bin_lake.sh: line 121: 22150 Segmentation fault: 11  l2bin infile=${INFILE} ofile=${ofile} prodtype=${prodtype} l3bprod=${3} resolve=${resolve} flaguse=${flaguse} noext=${noext}

When I open this file in SeaDAS there are clearly lots of valid latitudes (though some data are masked).  Does anyone know why this is happening and if it is the cause of the Segmentation fault:  11, whatever that is?  Is it possible that I am running out of memory or something?  This seems unlikely as the file is 3.6 mb.

I am using a Mac running OSX 10.11, 16GB ram, SeaDAS v7.4.  I've attached a png of the scene for reference and the script I am using for reference.

I have determined that processing the example file alone using the GUI is successful.  I have also determined, by using a different Mac with 24GB of ram, that this is not a memory issue (in my opinion).  Finally, it happens with flaguse ="NONE" or with a number of flags.

Thanks,
Dave
Attachment: l2bin_lake.sh - Shell script to process folder of L2 data (4k)
- By dshea Date 2017-05-18 10:56
Does l2bin fail the same way if you uncomment these lines in your script?

#export OCSSWROOT=/Applications/seadas-7.4/ocssw
#source ${OCSSWROOT}/OCSSW_bash.env
#echo ${OCSSWROOT}

This is how the GUI runs the processing programs.  I am assuming that SeaDAS 7.4 is installed in "/Applications/seadas-7.4"
- By Warner9392 Date 2017-05-18 10:58
Let me try that!
- By Warner9392 Date 2017-05-18 11:44
Yes, it fails the same way.  I made it through Rrs_412 without a Segmentation fault but there were still only 201 (of 207) scenes binned to L3 for this wavelength, and when I did Rrs_443, I got a single scene with a Segmentation fault and only 200 scenes binned.  Then I deleted the output for Rrs_443 and re-ran this wavelength and only got 199 binned files.

I can see getting a result that says "total_filled_bins: 0" for some scenes and NOT getting L3 output, but I just don't get why the number of scenes with this result would vary from wavelength to wavelength.  I also can't see why the results would change in repeated runs for the same wavelength. 

Perhaps there is something in the source code to help, but I have no idea how to read the source code.
Dave
- By gnwiii Date 2017-05-18 12:56
Like you, I was having network connection problems when installing the OCSSW Processing System on macOS El Capitan.  After installing, programs were failing with Segmentation fault: 11.  

In fact, if $OCSSWROOT/run/var/modis/leapsec.dat is missing, I get:
$ l2bin infile=$HOME/ocssw/benchmark/A2006167181000.L2_OC.nc ofile=A2006167181000.L3b_space.nc l3bprod="chlor_a" resolve=2 prodtype=regional
L2BIN 4.0.9 (Apr 27 2016 10:22:37)
INTERP parameter disabled.
Single HDF input
Averaging: standard
Resolution: 2
Max Qual Allowed: 2
prod_avg:
flagusemask: 107599675
required: 0
NetCDF4 input file
/Users/seadas/ocssw/benchmark/A2006167181000.L2_OC.nc   brk:    0   2030 167  65407
total number of bins: 95046854
1970001 2038018 270
krow:     0 out of   8640  (-90.00 to -84.38) Thu May 18 13:46:20 2017
[...]
krow:  5670 out of   8640  ( 28.12 to  33.75) Thu May 18 13:46:20 2017
Segmentation fault: 11


After running update_luts.py manually, the errors stopped.   I have also seen Segmentation fault errors where a user had multiple ocssw directories from failed or partial installs, e.g., after a failed install using the GUI (which defaults to /Applications/seadas-7.4/ocssw) they tried a command-line install which went to $HOME/ocssw.
- By Warner9392 Date 2017-05-18 14:17
Thanks gnwiii for yet another effort to help! 
I have

/Users/dmwarner/Applications/seadas-7.4/ocssw/run/var/modis/leapsec.dat

My script tells to use SeaDAS 7.4 and where it is and I checked for another instance of ocssw and don't see one.  I guess that is a good thing.

I ran update_luts.py aqua but still got Segmentation fault :11 AND different numbers of processing results for Rrs_412 and Rrs_443. 

I did not go back to running benchmark as that was the first thing I did and it seemed to work.  Maybe I should.
Dave
- By gnwiii Date 2017-05-19 10:15
Running the benchmark is helpful when you suspect a problem with your ocssw configuration.    The benchmark failed for me today  (at l1agen) on one El Capitan box after running update_luts.py, but started working after I ran the installer again (install_ocssw.py -v -b v7.4 --aqua).  My benchmark is modified to use the OC suite, So I ran the following stripped down version of your script:
#!/bin/bash
# mwe -- minimal working example for https://oceancolor.gsfc.nasa.gov/forum/oceancolor/topic_show.pl?pid=28692
#
export OCSSWROOT=${OCSSWROOT:-/Applications/seadas-7.4/ocssw}
echo ${OCSSWROOT}
source ${OCSSWROOT}/OCSSW_bash.env

resolve=1
flaguse="LAND,CLDICE,HISOLZEN,NAVFAIL,ATMFAIL"
prodtype=“regional”
noext=1
qual_max=1

infile=$OCSSWROOT/benchmark/A2006167181000.L2_OC.nc
for l3bprod in Rrs_412 Rrs_443 Rrs_469 Rrs_488 Rrs_531 Rrs_547 Rrs_555 Rrs_645 Rrs_667 Rrs_678 Rrs_748 chlor_a sst Kd_490 ; do
  ofile=A2006167181000.L3b_OC_$l3bprod.nc
  l2bin infile=${infile} ofile=${ofile} prodtype=${prodtype} l3bprod=${l3bprod} resolve=${resolve} flaguse=${flaguse} noext=${noext}
#  echo `rm -f $INFILE`
done


The "sst" product failed since it wasn't in the OC suite, but the rest appear to have worked.
- By dshea Date 2017-05-19 10:24
Find out which file with which band fails. Make sure that executing l2bin (without the script) fails.  Give me the input file and your exact command line arguments that fail and I will see what is going on.
- By Warner9392 Date 2017-05-19 10:26
gnwii
Thanks! 
I was and am working on creating a reproducible working example myself with just a subset of some of the files.  I should have done this before posting.  Thanks for not bashing me on this!

Not sure if this is part of the issue but even though I installed bash 4 through homebrew and thought I had it symlinked as my default shell, that seems not to have been successful as $ bash --version still tells me GNU bash, version 3.2.57(1)-release (x86_64-apple-darwin15).  Appears that I am having permission issues related to not having admin privileges.  Seems that if this were an issue I would not be able to use the gui for the task at hand.
- By Warner9392 Date 2017-05-19 11:01
dshea it seems I can't get l2bin to fail without the script on a file/band that generated a Segmentation fault :11 WITH the script.

I am not a programming wiz but I would say that means that there is something being caused by my script.

As a result I will attempt to rewrite the script as per suggestions from gnwiii.

Thanks for the effort to help!

Dave
- By dshea Date 2017-05-19 11:37
That is kind of strange.  I looked through your script for environment variables that might screw things up, but it looks fine to me.

don
- By Warner9392 Date 2017-05-19 11:55 Edited 2017-05-19 12:01
I can't see what is wrong with the code either.  I actually did not write it.  Another forum member blesht did.  He was kind enough to share it with me!

What is worse is I am only using the workflow I am using (L2 OC files > l2bin > L3b files > l3mapgen) because I can't figure out how to use gpt.command to reproject many files and save to files with all bands that I can use in R to calculate chlor_a with a Great Lakes-specific algorithm (R workflow uses raster package).  Would prefer, I think, to use L2 data anyway.

Maybe I need a new workflow.  Too bad as the existing R code works nicely. 

Thanks,
Dave
- By gnwiii Date 2017-05-19 12:39 Edited 2017-05-19 12:44
You can certainly encounter problems with environment settings when mixing shell programs.   There are some security issues with bash: CVE-2016-0634, CVE-2016-7543, CVE-2016-9401, CVE-2017-5932 were just mentioned in Ubuntu Security Notice USN-3294-1, but Apple is slow and doen't always tell us which CVE's are being patched so in addition to the new feaures in bash 4, security concerns may be reasons to use a recent version.

Have you seen Upgrade to bash 4 in macOS?   This illustrates a few ways to manage the shell version used in a particular script.    On recent versions of MacOS, Apple System Integrity Protection should prevent replacing /bin/bash with a symbolic link to /usr/local/bin/bash.   It is, however, possible to disable SIP.   The first line of your script reads "#!/bin/bash", so if you are using bash 4 in a terminal the script could be getting different settings.  From the examples in the "Upgrade to bash 4 " link you should be able to devise some scirpts to display key varaibles in
different shell environments.

If l2bin works from a shell prompt that points to some difference in the environment between the script and the prompt.  You can start from a very simple script and add back the extra parameter passing and checking logic  to see if you can figure out where things go wrong.
- By Warner9392 Date 2017-05-19 14:02
Thanks gnwii!  That link and is what I used to install bash 4 as far as I have.  It is installed but I've yet been able to make it the default shell.  Perhaps I can figure out how to make the script use bash 4.  I have been getting a message that says "/usr/local/bin/bash does not exist" in this process and I have no clue where homebrew installed bash 4 so that I can direct my script to it.

Dave
- By gnwiii Date 2017-05-19 14:49
Better to stick with the default Apple configuration until things are working since that is what most other people will have.  I haven't used homebrew in years, but I recall that the default was to install in a "private" area and create symbolic links under /usr/local.   There seems to something about your system that is "non-standard", either some missing or corrupt file in the ocssw directories or something (usually environment variable) causing the software to look for files in the wronge place.   That can easily happen if you have previousy installed seadas and forget to remove old environment settings when you install a newer version.  It has happened to many of us in my lab and can be tedious to resolve, even with the advantages of several pairs of eyes and side-by-side comparisons with a working system.
- By Warner9392 Date 2017-05-19 15:12
Thanks again!  Feeling worried now, more than anything else I suppose.

Not sure how I will get to the point of "things working".  And I wish I knew much earlier that issues with installation can have such far reaching impacts.  I would have taken greater care.

One thing that is nonstandard about my system is that it is a Mac.  That might be standard where you work, but is not where I work (USGS) and the implementation of security and other policies is not, in my opinion, well thought out even for Windows systems.  And the support for Macs, so far, has not been good.

This feature of my system, is, in all likelihood, part of the puzzle.

Do you mean that there is something causing ocssw to look for files in the wrong place?

In the meantime, I've modified my R code that uses L2 OC data and the raster and plotKML packages to estimate chlor_a so that it now also maps the estimated chlor_a.  While it is likely a different mapping procedure than is employed by OBPG products, it is still based on the principle of resampling and I can provide the code to anyone who wants to use it.  As long as the packages I used persist anyway.

Cheers,
Dave
- By gnwiii Date 2017-05-21 15:56
For every forum post like yours there are many other users who just give up and rely on the standard NASA products and most problems encountered by forum users show up sooner of later in our lab.

I suggest you set up a new "ocssw" account andt install using only the command-line (the default location will be ~/ocssw).  In the long run, having such a bare-bones account is often helpful for troubleshooting other issues. If you still having network problems, follow the offline installation instructions (download the git bundles and use the --local=<bundle_directory> option as well as the appropriate branch and sensor).  You  will need to use Anaconda python 2.7to run the installer and add lines to set the OCSSW variables to the "/Users/ocssw/.bashrc".  Logout and back in to get the settings in a terminal and run update_luts.py to make sure everything is current.  It is very likely that will get l2bin going, itf not the next step would be a basic install of macOS (e.g., on an external disk).

Once you have a configuration that runs l2bin without the segmentation faults, you can proceed with your processing and track down the difference at leisure.
- By Warner9392 Date 2017-05-21 16:37
I really appreciate you providing all the suggestions.  That sounds like a good plan to try.  Hopefully none of it will require admin authority or if required, it could be done in one sitting.  If it requires assistance from anyone with admin AND more experience than me then it could take a while. 

I do have R processing the stuff ok but slowly.  Something close to 30 seconds per scene with parallelization using foreach().
- By gnwiii Date 2017-05-23 07:44
Adding an ocssw user will require an admin, but only for a few minutes unless your site's policies require an extensive paperwork exercise for such accounts.
Up Topic SeaDAS / SeaDAS - General Questions / l2bin error

Powered by mwForum 2.29.7 © 1999-2015 Markus Wichitill