Not logged inOcean Color Forum

The forum is locked.

The Ocean Color Forum has transitioned over to the Earthdata Forum (https://forum.earthdata.nasa.gov/). The information existing below will be retained for historical reference. Please sign into the Earthdata Forum for active user support.

Up Topic SeaDAS / SeaDAS - General Questions / gpt merge (locked)
- By bruce Date 2016-10-21 13:34
Where is it documented?  With all of the available options...

I'm trying to add a new product to an existing file.  I can create the new product with out problem and if I use the Write operator it gets written on it's own to a new file but when I try and merge using the following node (basic structure from the help file and a couple online examples, NewProduct is what it's called in the bandMaths node)

  <node id="mergeNode">
    <operator>Merge</operator>
    <sources>
      <masterProduct>${source}</masterProduct>
      <sourceProducts>bandMathsNode</sourceProducts>
    </sources>
    <parameters>
      <includes>
  <include>
    <productId>NewProduct</productId>  
    <name>NewProduct</name>
    <newName>NewProduct</newName>
    <namePattern/>
  </include>
      </includes>
    </parameters>
  </node>

It fails with the ever helpful
java.lang.NullPointerException
  at org.esa.beam.gpf.operators.standard.MergeOp.copyBandWithFeatures(MergeOp.java:159)
  at org.esa.beam.gpf.operators.standard.MergeOp.initialize(MergeOp.java:116)
  at org.esa.beam.framework.gpf.internal.OperatorContext.initializeOperator(OperatorContext.java:457)
  at org.esa.beam.framework.gpf.internal.OperatorContext.getTargetProduct(OperatorContext.java:236)
  at org.esa.beam.framework.gpf.Operator.getTargetProduct(Operator.java:323)
  at org.esa.beam.framework.gpf.graph.NodeContext.initTargetProduct(NodeContext.java:74)
  at org.esa.beam.framework.gpf.graph.GraphContext.initNodeContext(GraphContext.java:195)
  at org.esa.beam.framework.gpf.graph.GraphContext.initNodeContext(GraphContext.java:178)
  at org.esa.beam.framework.gpf.graph.GraphContext.initOutput(GraphContext.java:162)
  at org.esa.beam.framework.gpf.graph.GraphContext.<init>(GraphContext.java:91)
  at org.esa.beam.framework.gpf.graph.GraphContext.<init>(GraphContext.java:64)
  at org.esa.beam.framework.gpf.graph.GraphProcessor.executeGraph(GraphProcessor.java:124)
  at org.esa.beam.framework.gpf.main.DefaultCommandLineContext.executeGraph(DefaultCommandLineContext.java:84)
  at org.esa.beam.framework.gpf.main.CommandLineTool.executeGraph(CommandLineTool.java:508)
  at org.esa.beam.framework.gpf.main.CommandLineTool.runGraph(CommandLineTool.java:356)
  at org.esa.beam.framework.gpf.main.CommandLineTool.runGraphOrOperator(CommandLineTool.java:249)
  at org.esa.beam.framework.gpf.main.CommandLineTool.run(CommandLineTool.java:150)
  at org.esa.beam.framework.gpf.main.CommandLineTool.run(CommandLineTool.java:122)
  at org.esa.beam.framework.gpf.main.GPT.run(GPT.java:54)
  at org.esa.beam.framework.gpf.main.GPT.main(GPT.java:34)
  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  at java.lang.reflect.Method.invoke(Method.java:497)
  at com.bc.ceres.launcher.Launcher.launch(Launcher.java:154)
  at com.bc.ceres.launcher.Launcher.main(Launcher.java:56)

Error: java.lang.NullPointerException
- By bruce Date 2016-10-21 14:41
OK, got it working (mostly)

This

  <node id="mergeNode">
    <operator>Merge</operator>
    <sources>
      <masterProduct>${source}</masterProduct>
      <sourceProducts>bandMathsNode</sourceProducts>
    </sources>
  </node>

works, but it dumps all of the data to "target.dim" and the directory "target.data".  which leads to 2 questions.  Is it possible to write it back into the L2_LAC_OC file?  and if not, how do I control the output filename? 

I have to say, while it's a powerful tool, the documentation (not a reflection on seadas, but rather on ESA and beam) "inhales briskly".
- By gnwiii Date 2016-10-23 10:47
Have you considered using beampy to add a product to level-2 NetCDF4-CF files?  Here is an example:


#! /usr/bin/env python3
# calc_spm_645.py -- add spm_645 to an existing level-2 file
# note that some processing programs require the new product to appear in the products.xml file
# uses Unidata netCDF4 module. To install:  "pip3 install --user netCDF4" 
import numpy as np
from netCDF4 import Dataset
import argparse

parser = argparse.ArgumentParser(description='Add spm_645 to Level-2 file with Rrs_645',
    usage='calc_spm_645 l2name', epilog='''
        Given a level-2 file with Rrs_645, apply the method of
        Nechad et al, RSE 2010, to compute spm_645 and add the
        band to the existing level-2 file.''')

parser.add_argument('l2name', nargs='+', help='name of the level-2 file')

args = parser.parse_args()
l2name = args.l2name[0]
nc=Dataset(l2name, 'a')
gd=nc.groups['geophysical_data']
Rrs_645=gd.variables['Rrs_645']

v = gd.createVariable('spm_645','f',('number_of_lines', 'pixels_per_line'),
                     fill_value=-999)
v.long_name = 'suspended particulate matter'
v.units = 'gm-3'
v.standard_name = 'spm_645'
v.valid_min = 0.01
v.valid_max = 1000.0
v.reference = 'Nechad, Ruddick, and Park, RSE 2010'
v[:] = 3.14*0.529*Rrs_645[:]
v[:] = 258.85*v[:]/(1-v[:]/0.1641)
nc.close()


Here are the changes to product.xml as a context diff:


*** product+spm.xml  2016-06-30 07:47:55.000000000 -0300
--- product.xml  2016-05-31 14:50:46.000000000 -0300
***************
*** 4621,4638 ****
              <description>Optimal SIOP combination, GIOP SVD_SIOP algorithm</description>
          </algorithm>
      </product>
-     <product name="spm">
-         <units>gm^-3</units>
-         <category>Derived</category>
-   <displayScale>log</displayScale>
-         <range>
-           <validMin>0.01</validMin>
-           <validMax>1000</validMax>
-         </range>
-         <algorithm name="spm">
-     <suffix>_645</suffix>
-           <cat_ix>999</cat_ix>
-           <description>Suspended particulate matter from 645 nm band</description>
-         </algorithm>
-     </product>
  </products>
- By bruce Date 2016-10-24 10:38
Thanks George...
A few questions.  I'm not a python wiz, and have run into a bit of a snag.  Any chance you can advise?
The new product equation depends on the value of 2 others...  In pseudo code.

     if old1 > 0 then new = old1 else if old2 > 1 then new = old2 else new=nan

v=np.where(old1>0,old1,old2) fails this way

Traceback (most recent call last):
  File "mytest.py", line 40, in <module>
    v=np.where(old1>0,old1,old2)
TypeError: __array__() takes no arguments (1 given)

Attempts at creating indices using
useold1=old1>0
useold2=old1<=0 and old2>1
v[useold1]=old1[useold1]
v[useold2]=old2[useold2]

fail thusly
Traceback (most recent call last):
  File "mytest.py", line 43, in <module>
    v[use2b]=old1[useold1]
  File "netCDF4.pyx", line 2900, in netCDF4.Variable.__getitem__ (netCDF4.c:35565)
MemoryError

Any thoughts? (I realize this is a python question, not a seadas question)
- By aynur Date 2016-10-24 15:30
To save the resulted file in a format other than "dim", you can use the "Write" operator after "Merge" and choose one of the "export" formats that are listed under File->Export in SeaDAS GUI. It's not possible to write it back into the L2_LAC_OC file yet.
- By gnwiii Date 2016-10-25 07:51
I'm no Python wizard, but I'm guessing that old1 and old2 are <type "netCDF4._netCDF4.Variable">.  You need to convert them to numpy.ndarray types.


$ python3.5
Python 3.5.2 (default, Oct 12 2016, 07:59:06)
[GCC 4.2.1 Compatible Apple LLVM 7.3.0 (clang-703.0.31)] on darwin
Type "help", "copyright", "credits" or "license" for more information.


>>> import numpy as np
>>> old1=np.array([0,1,2])
>>> type(old1)


<class 'numpy.ndarray'>

>>> old2 = 10+old1
>>> np.where(old1 >0, old1, old2)


array([10,  1,  2])


You may need to sprinkle some [:] incantations on your script.   See http://unidata.github.io/netcdf4-python/#section6
- By bruce Date 2016-10-25 13:44
got it working...  Posted here to help someone else in the future.  I don't know if the reshapes are strictly necessary, but it works as it is, so I'm happy.

#! /usr/bin/python
import sys
import numpy as np
from netCDF4 import Dataset
from copy import copy

l2name = sys.argv[1]
nc=Dataset(l2name, 'a')
no_lines = len(nc.dimensions[u'number_of_lines'])
no_pixels = len(nc.dimensions[u'pixels_per_line'])

gd=nc.groups['geophysical_data']
old1=gd.variables['old1']
old2=gd.variables['old2']

flat1 = np.reshape(old1,no_lines*no_pixels)
flat2 = np.reshape(old2,no_lines*no_pixels)

v = gd.createVariable('new','f',('number_of_lines', 'pixels_per_line'),
                      fill_value=-32767.0)
v.long_name = 'New algorithm'
v.units = 'units'
v.standard_name = 'new'
v.scale_factor=1.0
v.add_offset=0.0

tvar = np.where(flat1>0,flat1,np.where(flat2>0.1,flat2,-32767.0))
tvar = tvar.reshape(no_lines,no_pixels)
v.valid_min=np.nanmin(tvar)
v.valid_max=np.nanmax(tvar)
v[:]=tvar[:]

nc.close()
Up Topic SeaDAS / SeaDAS - General Questions / gpt merge (locked)

Powered by mwForum 2.29.7 © 1999-2015 Markus Wichitill