## GPT Cookbook: Workflow 3

## Reproject-Collocate-BandMaths
Subset-LandWater-StatisticsOp

Creates a new file with math band(s) based on an input source file with the option to retain source file bands in the new file.

|  |  |  | --> Write |  | --> Write |  | --> Write |  | --> Write |  | --> Write
Read |  | --> Reproject |  | --> Collocate |  | --> BandMaths |  | --> Subset |  | --> LandWater |  |
Read |  | --> Reproject |  | --> |  |  |  |  |  |  |  |
|  |  |  | --> Write |  |  |  |  |  | --> StatisticsOp |  |

This is a large example using a multitude of operators. It enables a comparison of two distinct level2 files.

    This example takes two input level2 files then:
        1. Reprojects each file applying quality masking and uses a Lambert Azimuthal Equal Area projection
        2. Collocates the reprojected files (with file1 treated as the reference crs)
        3. Adds two math bands (a difference band and ratio band of two desired primary bands)
        4  Subsets to reduce to only the two primary bands, plus the difference band and ratio band
        5. Adds land and water mask
        6. Statistics on the two bands, plus the difference band and ratio band
        7. Writes out the two reprojected files (NetCDF)
        8  Writes out the collocated file (NetCDF)
        9. Writes out the mathband file (NetCDF)
        10. Writes out the subset file (NetCDF)
        11. Writes out the landWater file (both NetCDF and BEAM-DIMAP*)

        * Masks do not get saved when writing out in formats other than BEAM-DIMAP

For more on the command line parameters used here see commented notes in "WorkflowEx03.par".

Note: Many of the parameters used in this example have the same value for each band and could each be consolidated into a single parameter. The separation of these parameters (using 'file1Type' and 'file2Type' instead of 'fileType') was done to create a more versatile example.

Note: This example could easily be adapted to compare aggregate files such as level-3 or Mosaic files.

Command Line Entry

For a comparison of sst within two distinct SST suite level2 files (a day and a night file).

    gpt WorkflowEx03.xml -p WorkflowEx03_SST.par -Pfile1=A2010283180500.L2_LAC_SST.nc -Pfile2=A2010283070000.L2_LAC_SST.nc -Pfile1Short=A20102831805 -Pfile2Short=A20102830700 -Pfile1Band=sst -Pfile2Band=sst -Phome=$HOME

... or for a comparison of chlor_a within two distinct OC suite level2 files.

    gpt WorkflowEx03.xml -p WorkflowEx03_OC.par -Pfile1=A2010283180500.L2_LAC_OC.nc -Pfile2=T2010282154500.L2_LAC_OC.nc -Pfile1Short=A20102831805 -Pfile2Short=T20102821545 -Pfile1Band=chlor_a -Pfile2Band=chlor_a -Phome=$HOME

... or similarly for a comparison of Kd_490 within two distinct OC suite level2 files.

    gpt WorkflowEx03.xml -p WorkflowEx03_OC.par -Pfile1=A2010283180500.L2_LAC_OC.nc -Pfile2=T2010282154500.L2_LAC_OC.nc -Pfile1Short=A20102831805 -Pfile2Short=T20102821545 -Pfile1Band=Kd_490 -Pfile2Band=Kd_490 -Phome=$HOME

File = WorkflowEx03.xml

    <graph id="WorkflowEx03Graph">
      <version>1.0</version>

        <node id="ReadNode1">
          <operator>Read</operator>
          <sources/>
          <parameters>
            <file>${file1}</file>
          </parameters>
        </node>

        <node id="ReadNode2">
          <operator>Read</operator>
          <sources/>
          <parameters>
            <file>${file2}</file>
          </parameters>
        </node>

        <node id="ReprojectNode1">
          <operator>Reproject</operator>
          <sources>
              <source>ReadNode1</source>
          </sources>

          <parameters>
              <crs>
                  PROJCS["Lambert_Azimuthal_Equal_Area / World Geodetic System 1984",
                         GEOGCS["World Geodetic System 1984",
                                DATUM["World Geodetic System 1984",
                                      SPHEROID["WGS 84", 6378137.0, 298.257223563, AUTHORITY["EPSG","7030"]],
                                AUTHORITY["EPSG","6326"]],
                         PRIMEM["Greenwich", 0.0, AUTHORITY["EPSG","8901"]],
                         UNIT["degree", 0.017453292519943295],
                         AXIS["Geodetic longitude", EAST],
                         AXIS["Geodetic latitude", NORTH]],
                  PROJECTION["Lambert_Azimuthal_Equal_Area"],
                  PARAMETER["latitude_of_center", ${lat}],
                  PARAMETER["longitude_of_center", ${lon}],
                  PARAMETER["false_easting", 0.0],
                  PARAMETER["false_northing", 0.0],
                  UNIT["m", 1.0],
                  AXIS["Easting", EAST],
                  AXIS["Northing", NORTH]]
              </crs>
              <resampling>${reprojectionResampling}</resampling>
              <orientation>0.0</orientation>
              <pixelSizeX>${pixelSize}</pixelSizeX>
              <pixelSizeY>${pixelSize}</pixelSizeY>
              <orthorectify>false</orthorectify>
              <noDataValue>NaN</noDataValue>
              <includeTiePointGrids>true</includeTiePointGrids>
              <addDeltaBands>false</addDeltaBands>
              <maskExpression>${file1ReprojectMaskExpression}</maskExpression>
              <retainValidPixelExpression>${retainValidPixelExpression}</retainValidPixelExpression>
              <applyValidPixelExpression>${applyValidPixelExpression}</applyValidPixelExpression>
          </parameters>

        </node>

        <node id="ReprojectNode2">
          <operator>Reproject</operator>
          <sources>
              <source>ReadNode2</source>
          </sources>

          <parameters>
              <crs>
                  PROJCS["Lambert_Azimuthal_Equal_Area / World Geodetic System 1984",
                         GEOGCS["World Geodetic System 1984",
                                DATUM["World Geodetic System 1984",
                                      SPHEROID["WGS 84", 6378137.0, 298.257223563, AUTHORITY["EPSG","7030"]],
                                AUTHORITY["EPSG","6326"]],
                         PRIMEM["Greenwich", 0.0, AUTHORITY["EPSG","8901"]],
                         UNIT["degree", 0.017453292519943295],
                         AXIS["Geodetic longitude", EAST],
                         AXIS["Geodetic latitude", NORTH]],
                  PROJECTION["Lambert_Azimuthal_Equal_Area"],
                  PARAMETER["latitude_of_center", ${lat}],
                  PARAMETER["longitude_of_center", ${lon}],
                  PARAMETER["false_easting", 0.0],
                  PARAMETER["false_northing", 0.0],
                  UNIT["m", 1.0],
                  AXIS["Easting", EAST],
                  AXIS["Northing", NORTH]]
              </crs>
              <resampling>${reprojectionResampling}</resampling>
              <orientation>0.0</orientation>
              <pixelSizeX>${pixelSize}</pixelSizeX>
              <pixelSizeY>${pixelSize}</pixelSizeY>
              <orthorectify>false</orthorectify>
              <noDataValue>NaN</noDataValue>
              <includeTiePointGrids>true</includeTiePointGrids>
              <addDeltaBands>false</addDeltaBands>
              <maskExpression>${file2ReprojectMaskExpression}</maskExpression>
              <retainValidPixelExpression>${retainValidPixelExpression}</retainValidPixelExpression>
              <applyValidPixelExpression>${applyValidPixelExpression}</applyValidPixelExpression>
          </parameters>

        </node>

        <node id="CollocateNode">
          <operator>Collocate</operator>
          <sources>
            <master>ReprojectNode1</master>
            <dependent>ReprojectNode2</dependent>
          </sources>
          <parameters>
              <targetProductType>COLLOCATED</targetProductType>
              <renameMasterComponents>${renameReference}</renameMasterComponents>
              <renameDependentComponents>${renameDependent}</renameDependentComponents>
              <masterComponentPattern>${ORIGINAL_NAME}_${file1Short}</masterComponentPattern>
              <dependentComponentPattern>${ORIGINAL_NAME}_${file2Short}</dependentComponentPattern>
              <resamplingType>${collocationResampling}</resamplingType>
          </parameters>
        </node>

        <node id="BandMathsNode">
          <operator>BandMaths</operator>
          <sources>
            <source>CollocateNode</source>
          </sources>
          <parameters>
            <copySourceFile>true</copySourceFile>
            <targetBands>

              <targetBand>
                <name>${file1Band}_${file1Short}_${file2Band}_${file2Short}_diff</name>
                <type>${diffBandType}</type>
                <expression>${file1Band}_${file1Short} - ${file2Band}_${file2Short}</expression>
                <description>${file1Band}_${file1Short} - ${file2Band}_${file2Short}</description>
                <unit>${diffBandUnit}</unit>
                <validExpression>${diffBandValidExpression}</validExpression>
                <noDataValue>${diffBandNoDataValue}</noDataValue>
              </targetBand>

              <targetBand>
                <name>${file1Band}_${file1Short}_${file2Band}_${file2Short}_ratio</name>
                <type>${diffBandType}</type>
                <expression>if (${file2Band}_${file2Short} != 0) then (${file1Band}_${file1Short} / ${file2Band}_${file2Short}) else NaN</expression>
                <description>if (${file2Band}_${file2Short} != 0) then (${file1Band}_${file1Short} / ${file2Band}_${file2Short}) else NaN</description>
                <unit>${diffBandUnit}</unit>
                <validExpression>${diffBandValidExpression}</validExpression>
                <noDataValue>${diffBandNoDataValue}</noDataValue>
              </targetBand>

            </targetBands>
          </parameters>
        </node>

      <node id="SubsetNode">
        <operator>Subset</operator>
        <sources>
            <source>BandMathsNode</source>
        </sources>

        <parameters>
            <bandNames>${file1Band}_${file1Short},${file2Band}_${file2Short},${file1Band}_${file1Short}_${file2Band}_${file2Short}_diff,${file1Band}_${file1Short}_${file2Band}_${file2Short}_ratio</bandNames>
        </parameters>

      </node>

      <node id="LandWaterMaskNode">
        <operator>LandWaterMask</operator>
        <sources>
          <source>SubsetNode</source>
        </sources>
        <parameters>
          <resolution>${resolution}</resolution>
          <superSamplingFactor>${superSamplingFactor}</superSamplingFactor>
        </parameters>
      </node>

        <node id="StatisticsOpNode">
          <operator>StatisticsOp</operator>
          <sources>
            <source>SubsetNode</source>
          </sources>
          <parameters>
            <bandConfigurations>
              <bandConfiguration>
                <sourceBandName>${file1Band}_${file1Short}</sourceBandName>
                <validPixelExpression>!nan(${file1Band}_${file1Short}_${file2Band}_${file2Short}_diff)</validPixelExpression>
              </bandConfiguration>
              <bandConfiguration>
                <sourceBandName>${file2Band}_${file2Short}</sourceBandName>
                <validPixelExpression>!nan(${file1Band}_${file1Short}_${file2Band}_${file2Short}_diff)</validPixelExpression>
              </bandConfiguration>
              <bandConfiguration>
                <sourceBandName>${file1Band}_${file1Short}_${file2Band}_${file2Short}_diff</sourceBandName>
              </bandConfiguration>
              <bandConfiguration>
                <sourceBandName>${file1Band}_${file1Short}_${file2Band}_${file2Short}_ratio</sourceBandName>
              </bandConfiguration>
            </bandConfigurations>
            <outputAsciiFile>${file1Short}_${file1Type}_${file2Short}_${file2Type}_${collocatedFileSuffix}_${bandMathsFileSuffix}_${subsetFileSuffix}_${statisticsFileSuffix}.txt</outputAsciiFile>
            <percentiles>${percentiles}</percentiles>
            <numBins>${numBins}</numBins>
            <calculateMedian>${calculateMedian}</calculateMedian>
          </parameters>
        </node>

        <node id="WriteReprojectNode1">
          <operator>Write</operator>
          <sources>
            <source>ReprojectNode1</source>
          </sources>
          <parameters>
            <file>${file1Short}_${file1Type}_${reprojectedFileSuffix}.${ofileExtension}</file>
            <formatName>${ofileFormat}</formatName>
          </parameters>
        </node>

        <node id="WriteReprojectNode2">
          <operator>Write</operator>
          <sources>
            <source>ReprojectNode2</source>
          </sources>
          <parameters>
            <file>${file2Short}_${file2Type}_${reprojectedFileSuffix}.${ofileExtension}</file>
            <formatName>${ofileFormat}</formatName>
          </parameters>
        </node>

        <node id="WriteCollocateNode">
          <operator>Write</operator>
          <sources>
            <source>CollocateNode</source>
          </sources>
          <parameters>
            <file>${file1Short}_${file1Type}_${file2Short}_${file2Type}_${collocatedFileSuffix}.${ofileExtension}</file>
            <formatName>${ofileFormat}</formatName>
          </parameters>
        </node>

        <node id="WriteBandMathsNode">
          <operator>Write</operator>
          <sources>
            <source>BandMathsNode</source>
          </sources>
          <parameters>
            <file>${file1Short}_${file1Type}_${file2Short}_${file2Type}_${collocatedFileSuffix}_${bandMathsFileSuffix}.${ofileExtension}</file>
            <formatName>${ofileFormat}</formatName>
          </parameters>
        </node>

        <node id="WriteSubsetNode">
          <operator>Write</operator>
          <sources>
            <source>SubsetNode</source>
          </sources>
          <parameters>
            <file>${file1Short}_${file1Type}_${file2Short}_${file2Type}_${collocatedFileSuffix}_${bandMathsFileSuffix}_${subsetFileSuffix}.${ofileExtension}</file>
            <formatName>${ofileFormat}</formatName>
          </parameters>
        </node>

        <node id="WriteLandWaterMaskNode">
          <operator>Write</operator>
          <sources>
            <source>LandWaterMaskNode</source>
          </sources>
          <parameters>
            <file>${file1Short}_${file1Type}_${file2Short}_${file2Type}_${collocatedFileSuffix}_${bandMathsFileSuffix}_${subsetFileSuffix}_${landWaterFileSuffix}.${ofileExtension}</file>
            <formatName>${ofileFormat}</formatName>
          </parameters>
        </node>

            <node id="WriteLandWaterMaskNodeFormat2">
          <operator>Write</operator>
          <sources>
            <source>LandWaterMaskNode</source>
          </sources>
          <parameters>
            <file>${file1Short}_${file1Type}_${file2Short}_${file2Type}_${collocatedFileSuffix}_${bandMathsFileSuffix}_${subsetFileSuffix}_${landWaterFileSuffix}.${ofileExtension2}</file>
            <formatName>${ofileFormat2}</formatName>
          </parameters>
        </node>

    </graph>

File = WorkflowEx03_SST.par

    # Read operator parameters (commented out parameters are optionally being sent in at command line)
    # file1=A2010283180500.L2_LAC_SST.nc  -- this is the input file which will act as the reference/master in the collocation step
    # file2=A2010283070000.L2_LAC_SST.nc  -- this input file will act as the dependent in the collocation step
    ifileExtension=nc

    # Write operator parameters (commented out parameters are optionally being sent in at command line)
    # file1Short=A283180500  -- this name gets used as the suffix for the bands collocated from file1, and for constructing the collocated output file name
    # file2Short=A283070000  -- this name gets used as the suffix for the bands collocated from file2, and for constructing the collocated output file name
    # file1Type and file2Type is used in constructing the output file names
    # These 4 parameters are unique to SST products.  To compare OC or IOP files you would need to alter these 4 parameters
    file1Type=L2_LAC_SST
    file2Type=L2_LAC_SST
    #file1Band=sst
    #file2Band=sst

    reprojectedFileSuffix=Reprojected
    collocatedFileSuffix=Collocated
    bandMathsFileSuffix=BandMaths
    subsetFileSuffix=Subset
    landWaterFileSuffix=LandWater

    ofileFormat=NetCDF-CF
    ofileExtension=nc
    ofileFormat2=BEAM-DIMAP
    ofileExtension2=dim

    # Reproject operator parameters
    lat=37.0
    lon=-76.0
    pixelSize=1000.0
    reprojectionResampling=Bilinear
    retainValidPixelExpression=true
    applyValidPixelExpression=true

    # The reprojectMaskExpression is unique to SST products and flags.  To compare OC or IOP files you would need to alter this expression
    file1ReprojectMaskExpression=(qual_sst == 0 or qual_sst == 1) and !l2_flags.HISATZEN
    file2ReprojectMaskExpression=(qual_sst == 0 or qual_sst == 1) and !l2_flags.HISATZEN

    # Collocate operator parameters
    renameReference=true
    renameDependent=true
    collocationResampling=NEAREST_NEIGHBOUR

    # BandMaths operator parameters
    diffBandType=float64
    diffBandUnit=Celsius
    diffBandValidExpression=
    diffBandNoDataValue=NaN

    # LandWaterMask operator parameters
    resolution=1000
    superSamplingFactor=3

    # StatisticsOp operator parameters
    percentiles=80,85,90
    numBins=1000
    calculateMedian=true
    statisticsFileSuffix=Statistics

    # WriteImage operator parameters
    writeImageFormat=png
    cpdFilePath=.seadas/beam-ui/auxdata/color-palettes/anomalies1_universal.cpd
    cpdAutoDistribute=false
    colourScaleMin=-4.0
    colourScaleMax=4.0
    level=0
    scaleType=linear

File = WorkflowEx03_OC.par

    # Read operator parameters (commented out parameters are optionally being sent in at command line)
    # file1=A2010283180500.L2_LAC_SST.nc  -- this is the input file which will act as the reference/master in the collocation step
    # file2=A2010283070000.L2_LAC_SST.nc  -- this input file will act as the dependent in the collocation step
    ifileExtension=nc

    # Write operator parameters (commented out parameters are optionally being sent in at command line)
    # file1Short=A283180500  -- this name gets used as the suffix for the bands collocated from file1, and for constructing the collocated output file name
    # file2Short=A283070000  -- this name gets used as the suffix for the bands collocated from file2, and for constructing the collocated output file name
    # file1Type and file2Type is used in constructing the output file names
    # These 4 parameters are unique to OC products.  To compare SST or IOP files you would need to alter these 4 parameters
    file1Type=L2_LAC_OC
    file2Type=L2_LAC_OC
    #file1Band=chlor_a
    #file2Band=chlor_a

    reprojectedFileSuffix=Reprojected
    collocatedFileSuffix=Collocated
    bandMathsFileSuffix=BandMaths
    subsetFileSuffix=Subset
    landWaterFileSuffix=LandWater

    ofileFormat=NetCDF-CF
    ofileExtension=nc
    ofileFormat2=BEAM-DIMAP
    ofileExtension2=dim

    # Reproject operator parameters
    lat=37.0
    lon=-76.0
    pixelSize=1000.0
    reprojectionResampling=Bilinear
    retainValidPixelExpression=true
    applyValidPixelExpression=true

    # The reprojectMaskExpression is unique to OC products and flags.  To compare SST files you would need to alter this expression
    file1ReprojectMaskExpression=!l2_flags.HIGLINT and !l2_flags.HISATZEN and !l2_flags.LOWLW and !l2_flags.HILT
    file2ReprojectMaskExpression=!l2_flags.HIGLINT and !l2_flags.HISATZEN and !l2_flags.LOWLW and !l2_flags.HILT

    # Collocate operator parameters
    renameReference=true
    renameDependent=true
    collocationResampling=NEAREST_NEIGHBOUR

    # BandMaths operator parameters
    diffBandType=float64
    diffBandUnit=Celsius
    diffBandValidExpression=
    diffBandNoDataValue=NaN

    # LandWaterMask operator parameters
    resolution=1000
    superSamplingFactor=3

    # StatisticsOp operator parameters
    percentiles=80,85,90
    numBins=1000
    calculateMedian=true
    statisticsFileSuffix=Statistics

    # WriteImage operator parameters
    writeImageFormat=png
    cpdFilePath=.seadas/beam-ui/auxdata/color-palettes/anomalies1_universal.cpd
    cpdAutoDistribute=false
    colourScaleMin=-4.0
    colourScaleMax=4.0
    level=0
    scaleType=linear