Lo all,

I was processing some recent data (4 different areas) that all rely on the same anc data I believe. They all errored out like this:

Loading radiance to brightness temperature from /optics1/software/seadas/seadas-7.5/ocssw/share/modis/aqua/cal/bt_modisa.hdf
Loading land mask file from /optics1/software/seadas/seadas-7.5/ocssw/share/common/landmask_GMT15ARC.nc
Loading bathymetry mask file from /optics1/software/seadas/seadas-7.5/ocssw/share/common/watermask.dat
Loading ice mask file from /optics1/software/seadas/seadas-7.5/ocssw/var/anc/2019/218/N201921800_SEAICE_NSIDC_24h.hdf
Loaded near real time NSIDC ice HDF file.
Loading elevation file from /optics1/software/seadas/seadas-7.5/ocssw/share/common/ETOPO1_ocssw.nc
Loading XCAL rvs and polarization sensitivities from /optics1/software/seadas/seadas-7.5/ocssw/var/modisa/xcal/OPER/xcal_modisa_axc_oc_v1.30d_412.hdf
Loading XCAL rvs and polarization sensitivities from /optics1/software/seadas/seadas-7.5/ocssw/var/modisa/xcal/OPER/xcal_modisa_axc_oc_v1.30d_443.hdf
Loading XCAL rvs and polarization sensitivities from /optics1/software/seadas/seadas-7.5/ocssw/var/modisa/xcal/OPER/xcal_modisa_axc_oc_v1.30d_469.hdf
Loading XCAL rvs and polarization sensitivities from /optics1/software/seadas/seadas-7.5/ocssw/var/modisa/xcal/OPER/xcal_modisa_axc_oc_v1.30d_488.hdf
Loading XCAL rvs and polarization sensitivities from /optics1/software/seadas/seadas-7.5/ocssw/var/modisa/xcal/OPER/xcal_modisa_axc_oc_v1.30d_531.hdf
Loading XCAL rvs and polarization sensitivities from /optics1/software/seadas/seadas-7.5/ocssw/var/modisa/xcal/OPER/xcal_modisa_axc_oc_v1.30d_547.hdf
Loading XCAL rvs and polarization sensitivities from /optics1/software/seadas/seadas-7.5/ocssw/var/modisa/xcal/OPER/xcal_modisa_axc_oc_v1.30d_555.hdf
Loading XCAL rvs and polarization sensitivities from /optics1/software/seadas/seadas-7.5/ocssw/var/modisa/xcal/OPER/xcal_modisa_axc_oc_v1.30d_645.hdf
Loading XCAL rvs and polarization sensitivities from /optics1/software/seadas/seadas-7.5/ocssw/var/modisa/xcal/OPER/xcal_modisa_axc_oc_v1.30d_667.hdf
Loading XCAL rvs and polarization sensitivities from /optics1/software/seadas/seadas-7.5/ocssw/var/modisa/xcal/OPER/xcal_modisa_axc_oc_v1.30d_678.hdf
Loading XCAL rvs and polarization sensitivities from /optics1/software/seadas/seadas-7.5/ocssw/var/modisa/xcal/OPER/xcal_modisa_axc_oc_v1.30d_748.hdf
Loading XCAL rvs and polarization sensitivities from /optics1/software/seadas/seadas-7.5/ocssw/var/modisa/xcal/OPER/xcal_modisa_axc_oc_v1.30d_869.hdf
Loading DEM info from /optics1/software/seadas/seadas-7.5/ocssw/share/common/ETOPO1_ocssw.nc
-E- /home/seadas/ocssw/src/l2gen/sstref.c : Unable to initialize SST file
/optics1/software/seadas/seadas-7.5/ocssw/var/anc/2019/212/N2019212_SST_OIV2AV_24h.nc 219

So I have deleted that offending N2019212_SST_OIV2AV_24h.nc file and run the processing again, and it downloads the N2019212_SST_OIV2AV_24h.nc file again (exactly the same), but the same failures. Is that file corrupt on your end?

Please advise?

Thanks,
Brock

The issue is that when NOAA restarted the process ('twas down for a bit due to a METOPS source problem), the title attribute changed from what was previously written.
Since l2gen uses that string to select the function to read the file, it now fails.

Until we can update l2gen, a quick and dirty fix is to change the tilte back to the expected one:
:title = "NCEI Daily-OISST-V2 based mainly on AVHRR, Final" ;

There should be a script in $OCSSWROOT/scripts called ncattredit.py that can help you do that.

Sean

Does this mean that all processing of data that requires AVHRR SST post 2019/212 will now fail until l2gen is updated? If that is the case, how long might this be? Much of my work is real-time and automated. It might be difficult to find a place to call ncattredit.py in the current stream.

Brock

BTW, all my previous N2019211 and lower had :

:title = "NCEI Daily-OISST-V2 based mainly on AVHRR, Interim" ;

So I changed it to that, and was able to reprocess the data.

Brock

> Does this mean that all processing of data that requires AVHRR SST post 2019/212 will now fail until l2gen is updated? If that is the case, how long might this be?

Yes, unless NOAA reverts the title string.  I'm out of the office until August 14th, so at least until then, if not longer.

The "Interim" version gets replaced by the "Final" version a week or two later.  We process with the interim and reprocess when the final arrive.

Sean

Sean,
Any news on the post 2019/212 N20192??_SST_OIV2AV_24h.nc files yet? Or an update to l2gen? I am still having to manually convert them daily.
Cheers,
Brock

l2gen has been patched and the post 2019/212 OI2AV SST files are available for download.

Tommy

Tommy,

Can you please advise on the best way to update "just" l2gen?
https://oceancolor.gsfc.nasa.gov/docs/ocssw/md_src_l2gen_Changelog.html

Shows
Unreleased
9.3.0 - 2018-04-09

However, I did a install to another location, and then tried to run the new version which ran. I see there is a large difference in the size of the l2gen file. Is that correct? WIll replacing this file alone work?

[bmurch@optics0 bin]$ pwd; ll ./l2gen /shares/cms_optics/apps/seadas/seadas-7.5/ocssw/bin/l2gen
/shares/cms_optics/apps/seadas-7.5.3/bin
-rwxrwxr-x 1 bmurch cms_optics 4418424 Aug 23 16:46 ./l2gen
-rwxrwxr-x 1 bmurch cms_optics 5178400 Mar 14 10:18 /shares/cms_optics/apps/seadas/seadas-7.5/ocssw/bin/l2gen

Thanks,
Brock

No, simply updating the l2gen binary will not work.  The new binaries use shared libraries (well more of them anyway) and so will not work unless you also updated those libraries. BTW, I have not had time to update the externally accessible doxygen pages (in part because I've not updated the externally accessible source) ...on the todo list...it's a long list...

You could try running ldd on the l2gen binary and make sure your $OCSSWROOT/opt/lib directory contains the necessary libraries.

Sean

Sean,
Thanks for your advise, I will check. It appears that these are missing:

[bmurch@optics0 bin]$ ldd l2gen_new_753  | grep found
        liboc.so => not found
        libhdf5_hl.so.10.3.0 => not found
        libhdf5.so.10.3.0 => not found
        libboost_filesystem.so.1.69.0 => not found
        libboost_date_time.so.1.69.0 => not found
        libjansson.so.4 => not found

Is it possible to just drop these into the proper opt opt directory? How will that mess up the git?
Or back to the original question of: Is there a way to simply install only what l2gen needs?

Cheers,
Brock

Brock,
It should be fine to drop the missing libraries into the opt/lib directory....make sure that all the links are correct.
For example, that libhdf5_hl.so.10.3.0 is a link :libhdf5_hl.so.10.3.0 -> libhdf5_hl.so.1.8.19

Sean

Sean,
I think I got everything done correctly eg:
[optics0]$ ldd /shares/cms_optics/apps/seadas/seadas-7.5/ocssw/bin/l2gen_new_753 | grep found
        liboc.so => not found                                                                                   
        libhdf5_hl.so.10.3.0 => not found                                                                       
        libhdf5.so.10.3.0 => not found                                                                          
        libboost_filesystem.so.1.69.0 => not found                                                              
        libboost_date_time.so.1.69.0 => not found                                                               
        libjansson.so.4 => not found
[optics0]$ ldd /shares/cms_optics/apps/seadas/seadas-7.5/ocssw/bin/l2gen_new_753 | grep found
[optics0]$

However, I noticed that these two libboost_filesystem.so.1.69.0 and libboost_system.so.1.69.0

which have different links in the new version eg:
new
[optics0]$ ll /shares/cms_optics/apps/seadas-7.5.3/opt/lib/libboost_filesystem.so
lrwxrwxrwx 1 bmurch cms_optics 29 Aug 23 16:46 /shares/cms_optics/apps/seadas-7.5.3/opt/lib/libboost_filesystem.so -> libboost_filesystem.so.1.69.0
vs old
[optics0]$ ll /shares/cms_optics/apps/seadas/seadas-7.5/ocssw/opt/lib/libboost_filesystem.so
lrwxrwxrwx 1 bmurch cms_optics 29 Jun  5  2018 /shares/cms_optics/apps/seadas/seadas-7.5/ocssw/opt/lib/libboost_filesystem.so -> libboost_filesystem.so.1.60.0

Are not called directly by l2gen by their links (libboost_system.so && libboost_filesystem.so) so I don't know if it necessary (or dangerous) to unlink libboost_system.so && libboost_filesystem.so from the current 1.60.0 to create new links to the 1.69.0 versions?

Please let me know if there are different things linking to libboost_system.so && libboost_filesystem.so as I won't change the link in those cases. I'm pretty sure that others are linking to those, as they are not even in the old l2gen (but still in the opt/lib dir}
eg:
[optics0]$ ldd ./l2gen | grep libboost
[optics0]$

[bmurch@optics0 lib]$ ln -s libboost_filesystem.so.1.69.0 libboost_filesystem.so
[bmurch@optics0 lib]$ ln: creating symbolic link `libboost_filesystem.so': File exists
[bmurch@optics0 lib]$ ll libboost_system.so
lrwxrwxrwx 1 bmurch cms_optics 25 Jun  5  2018 libboost_system.so -> libboost_system.so.1.60.0
[bmurch@optics0 lib]$ ll libboost_system.so*
lrwxrwxrwx 1 bmurch cms_optics    25 Jun  5  2018 libboost_system.so -> libboost_system.so.1.60.0
-rwxrwxr-x 1 bmurch cms_optics 17040 Jun  5  2018 libboost_system.so.1.60.0
-rwxrwxr-x 1 bmurch cms_optics  8048 Aug 24 22:50 libboost_system.so.1.69.0
[bmurch@optics0 lib]$ ll libboost_filesystem.so*
lrwxrwxrwx 1 bmurch cms_optics     29 Jun  5  2018 libboost_filesystem.so -> libboost_filesystem.so.1.60.0
-rwxrwxr-x 1 bmurch cms_optics 224560 Jun  5  2018 libboost_filesystem.so.1.60.0
-rwxrwxr-x 1 bmurch cms_optics 250752 Aug 24 22:48 libboost_filesystem.so.1.69.0

Cheers
Brock

Sean,

I now seeing this error (-E- clo_getOptionString: option=water_spectra_file needs to be set), please advise:

[optics0]$ /shares/cms_optics/apps/seadas/seadas-7.5/ocssw/bin/l2gen_new_9.5.0-C2019.3  par=A20192381530.1KM.NAAMES.L2_LAC_OC.par
Input file /work/b/bmurch/tmp/S4P_MODIS_H5_MODISA.NAAMES.2019.238_version_0.08_HwTi_97B_copy/MYD021KM.2019238.1530.NAAMES.hdf is MODIS Aqua Level-1B HDF-EOS product.
Loading default parameters from /shares/cms_optics/apps/seadas/seadas-7.5/ocssw/share/common/msl12_defaults.par
Loading default parameters for MODISA from /shares/cms_optics/apps/seadas/seadas-7.5/ocssw/share/modis/msl12_defaults.par
Loading default sub-sensor parameters for MODISA from /shares/cms_optics/apps/seadas/seadas-7.5/ocssw/share/modis/aqua/msl12_defaults.par
Loading parameters for suite OC from /shares/cms_optics/apps/seadas/seadas-7.5/ocssw/share/modis/msl12_defaults_OC.par
Loading command line parameters

Loading user parameters for MODISA

Loading characteristics for MODISA
Opening sensor information file /shares/cms_optics/apps/seadas/seadas-7.5/ocssw/share/modis/aqua/msl12_sensor_info.dat
  Bnd   Lam       Fo    Tau_r     k_oz    k_no2    t_co2   awhite       aw      bbw
    0  412.000  172.912 3.099e-01 1.987e-03 5.814e-19 1.000e+00 1.000e+00 2.030e-02 2.884e-03
    1  443.000  187.622 2.367e-01 3.189e-03 4.985e-19 1.000e+00 1.000e+00 1.110e-02 2.181e-03
    2  469.000  205.878 1.914e-01 8.745e-03 3.938e-19 1.000e+00 1.000e+00 1.034e-02 1.749e-03
    3  488.000  194.933 1.592e-01 2.032e-02 2.878e-19 1.000e+00 1.000e+00 1.676e-02 1.447e-03
    4  531.000  185.747 1.126e-01 6.838e-02 1.525e-19 1.000e+00 1.000e+00 4.494e-02 1.014e-03
    5  547.000  186.539 9.906e-02 8.622e-02 1.194e-19 1.000e+00 1.000e+00 5.491e-02 8.886e-04
    6  555.000  183.869 9.432e-02 9.553e-02 9.445e-20 1.000e+00 1.000e+00 5.839e-02 8.443e-04
    7  645.000  157.811 5.082e-02 7.382e-02 1.382e-20 1.000e+00 9.133e-01 3.458e-01 4.469e-04
    8  667.000  152.255 4.443e-02 4.890e-02 7.065e-21 1.000e+00 8.921e-01 4.923e-01 3.900e-04
    9  678.000  148.052 4.146e-02 3.787e-02 8.304e-21 1.000e+00 8.783e-01 5.116e-01 3.634e-04
   10  748.000  128.065 2.849e-02 1.235e-02 2.157e-21 1.000e+00 7.832e-01 2.800e+00 2.470e-04
   11  859.000   97.174 1.613e-02 2.347e-03 6.212e-23 1.000e+00 6.519e-01 4.396e+00 1.384e-04
   12  869.000   95.824 1.540e-02 1.936e-03 7.872e-23 1.000e+00 6.403e-01 4.727e+00 1.320e-04
   13 1240.000   45.467 3.617e-03 0.000e+00 0.000e+00 9.994e-01 0.000e+00 1.147e+02 3.021e-05
   14 1640.000   23.977 1.219e-03 0.000e+00 0.000e+00 9.890e-01 0.000e+00 6.689e+02 1.000e-05
   15 2130.000    9.885 4.286e-04 0.000e+00 0.000e+00 9.696e-01 0.000e+00 2.810e+03 3.465e-06

  Bnd   Lam
   16 3750.000
   17 3959.000
   18 4050.000
   19 6715.000
   20 7325.000
   21 8550.000
   22 11000.000
   23 12000.000

-E- clo_getOptionString: option=water_spectra_file needs to be set

Update the data files:

~/ocssw/share/common$ git diff ^HEAD~1 msl12_defaults.par                                     
diff --git a/msl12_defaults.par b/msl12_defaults.par                                                                   
index a019bf2..cbf0e4e 100644                                                                                          
--- a/msl12_defaults.par                                                                                               
+++ b/msl12_defaults.par                                                                                               
@@ -53,6 +53,9 @@  tauafile=$OCDATAROOT/common/taua865_climatology.hdf
# for brdf                                                                                                             
fqfile=$OCDATAROOT/common/morel_fq.nc                                                                                
+# for aw/bbw                                                                                                          
+water_spectra_file=$OCDATAROOT/common/water_spectra.dat                                                              
+                                                                                                                       
# lon/lat subsampling                                                                                                 
# -----------------------                                                                                              
ctl_pt_incr=1