SeaWiFS Reprocessing #4


Over the past few months, the SeaWiFS Project in collaboration with members of the ocean color research community has conducted a thorough review of the calibration, algorithms and operational procedures used to process SeaWiFS data. Significant progress has been made in a large number of areas which we believe greatly improve the scientific quality of the SeaWiFS data products and justify the consideration of a complete mission reprocessing (September 1997 through present) and the incorporation of those changes into the operational processing stream. However, before implementing any changes to the operational products that are archived and distributed by the Goddard DAAC, we have conducted a very comprehensive set of evaluations for each of the proposed changes.

The SeaWiFS Project has now completed these evaluations and we are in the process of organizing and posting the results along with descriptions of all the changes that were considered and would like to request that all interested members of the ocean color research community review these results. We solicit your comments, questions and suggestions as part of our final evaluation process prior to the reprocessing. We realize that science is a continuous process and that there will always be additional improvements that can be made as our understanding improves. However, we believe that the results posted clearly demonstrate that we are able to produce better products today than we were two years ago when the last SeaWiFS reprocessing was carried out. Based on a number of complete mission reprocessings that were done as a part of our evaluations, we anticipate being able to reprocess all the standard archive products for the entire mission in less than one week.

Presented below is a summary of the results for each of the detailed incremental tests that were run to assess the effect of each change proposed for SeaWiFS Reprocessing #4. These tests were all run for the month of May 1999. The summary for each test includes a statement of the changes added for that test, and a brief discussion of the changes in the products which resulted from that test. The discussions include links to text, images and plots on the web which describe the changes in detail and illustrate the results. In addition to these detailed tests, additional analyses were performed on the complete SeaWiFS mission period and these results are presented as well.

A summary of all the changes considered for reprocessing #4 can be found here.

NOTE: All links on this page will be displayed in a second browser window.

Reprocessing #3 Global Chlorophyll image

Baseline - Current Operational Processing

(ALGTEST 0)

In May 2000, the SeaWiFS Project after extensive review and community involvement initiated Reprocessing #3 and incorporated those changes into the operational processing stream. This data set serves as the baseline from which the proposed changes were compared and a comparison between Reprocessing #3 and the proposed Reprocessing #4 is presented below.

Case 1: MOBY Straylight Correction

(ALGTEST 1)
Scatter Plot of Chlorophyll - Case1:Reprocessing #3

This case incorporates the new visible-band vicarious gains which are derived from the MOBY data corrected for straylight effects, along with refinements to the vicarious calibration procedures The configuration is otherwise identical to that for Reprocessing #3.

The effect of the MOBY correction is to increase the gains significantly in Bands 1 and 2 and slightly in Band 3, and to decrease Bands 4 and 5; the procedural changes increased all of the gains, with larger increases in the green bands. The results of the gain changes are increased nLw in all visible bands. The comparison of results indicate that the median increase in nLw for bands 1 and 2 were 17% and 9%, respectively, with smaller increases in the other bands (6% in 490, 3% in 510, and 5% in 555). These changes resulted in a 6% decrease in the median global chlorophyll.

The bin coverage increased by 0.6% and the number of samples increased by 2.4% for the month of May 1999. The main cause for the increase was a reduction of samples flagged for absorbing aerosol (59% of the increase). The increase in Band 1 nLw reduced the absorbing aerosol index and thus, fewer pixels were designated as having excessive absorbing aerosols. The absorbing aerosol index (described in Chapter 1 of Vol 10 of the Post Launch TM series) is sensitive to the calibration of Band 1. Most of the remaining increased samples were due to reduced flagging for low Band 5 nLw (an indicator of cloud shadows), due to the small increase in the Band 5.

Summary Statstics: Median Percent Change from Operational (Reprocessing #3)
Filled Bins Samples/Bin chlor_a nLw(412) nLw(490) nLw(555) tau(865)
Monthly Daily Monthly Daily Monthly Daily Monthly Daily Monthly Daily Monthly Daily Monthly Daily
+0.6 +1.9 +2.4 +2.8 -5.96 -5.23 +16.9 +16.5 +6.20 +6.20 +4.74 +4.77 +0.00 +0.00

Case 2: Updated Calibration Table

(ALGTEST 2)
Scatter Plot of Chlorophyll - Case2:Case1

This case incorporates a new calibration table which implements changes in the calibration time dependence (from piecewise quadratic/linear to exponential), temperature corrections and mirror-side corrections for all bands. The change also includes a re-evaluation of the vicarious gains in Bands 1 through 7, and an adjustment to the cloud mask threshold to compensate for a reduction in the Band 8 calibrated radiance.

The main effect of this change is an overall reduction in 865 nm AOT of about 3%, caused by the change in the Band 8 time-dependent calibration, which reduces the TOA radiances throughout the mission. This also results in loss of samples, primarily due to the increase in the atmospheric warning flag (ATMWARN), causing 44.9% of the loss in samples followed by the low water-leaving radiance at 555 nm (LOWLW) flag, causing 24.5% of the loss. The exponential calibration slightly lowered the band 7 and 8 total radiances. This caused some of the ratios of aerosol radiances in band 7 and 8, or epsilon, to occasionally get higher and lower than the acceptable bounds, triggering the atmospheric warning flag. The low epsilon limit was predominently exceeded in more cases. The band 5 nLw decreased slightly, causing an increase in flagging due to the low nLw at 555 nm (LOWLW) flag. The gain, of samples was almost all caused by a decrease in flagging by the stray light (STRAYLIGHT) flag. As the band 8 radiance was lowered slightly, fewer samples were considered a bright target which would require stray light flagging.

The nLw's are also affected, due to a combination of changes to the calibration time dependences, vicarious calibration and AOT. In the May 1999 time frame, the nLw's are reduced by 1 - 2% on average. Chlorophyll for the same time period, increased by approximately 1% globally over the previous case. The results will not be the same throughout the mission, because of the change to the functional form of the time dependences.

Summary Statstics: Median Percent Change from Case 1
Filled Bins Samples/Bin chlor_a nLw(412) nLw(490) nLw(555) tau(865)
Monthly Daily Monthly Daily Monthly Daily Monthly Daily Monthly Daily Monthly Daily Monthly Daily
-0.1 -0.4 -0.5 -1.0 +1.11 +1.06 -1.83 -1.73 -1.41 -1.33 -0.97 -0.89 -3.27 -3.38
Scatter Plot of Chlorophyll - Case3:Case2

Case 3: Revised Out-of-Band Correction

(ALGTEST 9 aka ALGTEST2.1)

This case incorporates the revised out-of-band correction that is applied prior to use in downstream computations (e.g., chlorophyll retrieval). The correction is computed from the Band 3/Band 5 nLw ratio, and is based on a chlorophyll-dominated nLw spectrum. A revised set of correction factors were generated using the recently published clear-water reflectance model by Morel and Maritorena (2001).

The change to the correction factors result in a lowering of chlorophyll values in waters with a Band 3/Band 5 nLw ratio of >~ 2. For lower ratios, chlorophyll increases slightly. This can be clearly seen in the test results. The difference and ratio images with respect to test 2 illustrate the decrease in low chlorophyll values, particularly in the central gyres, and a slight increase in high productivity waters. Globally, the effect is on the order of a 2.5 - 4% decrease in chlorophyll over the previous test.

Summary Statstics: Median Percent Change from Case 2
Filled Bins Samples/Bin chlor_a nLw(412) nLw(490) nLw(555) tau(865)
Monthly Daily Monthly Daily Monthly Daily Monthly Daily Monthly Daily Monthly Daily Monthly Daily
-0.0 -0.2 -0.3 -0.3 -3.70 -3.81 +2.26 +2.26 -0.22 -0.21 -1.30 -1.35 +0.00 +0.00

Case 4: Miscellaneous Changes and Corrections

(ALGTEST 3)
Scatter Plot of Chlorophyll - Case4:Case3

This case incorporates several miscellaneous changes. The high LT flag, which was previously set when any band exceeded the bilinear calibration "knee" value, was changed to only test the NIR bands, where the increased quantization of the radiances can significantly impact aerosol retrievals. The high Lt flag is applied as a mask at Level-3. In addition, the threshold on high view zenith angle was increased from 56 to 60-deg. A new turbid water flag was introduced, but this flag is only informational. At this stage in the test sequence, we also introduce the improved handling of low aerosol conditions, a modified NIR iteration control, and the vanishing NIR correction. In addition, the transmittance in the cloud albedo calculation was modified to improve behavior at high solar and view angles, and the NIR water-leaving radiance model was updated with new absorption and backscatter coefficients. The cloud albedo threshold was adjusted to compensate for the algorithm change. The cloud albedo changes were superseded by subsequent modifications (Case 7).

The most notable impact of these miscellaneous changes is a general (3%) increase in samples per bin, due to the relaxed HILT and HISATZEN masking and the change in the cloud albedo algorithm.The greatest amount of sample gain (48%) was due to the relaxing of the high satellite zenith cutoff (HISATZEN). A change in the albedo computation accounted for 17% of the sample gain (CLDICE flag). The adoption of a better treatment of very low NIR aerosol radiances resulted in the 11% gain in samples (ATMWARN flag). The band 5 nLw slightly decreased in some regions causing a gain in samples of 10% due to decreased occurence in the low water-leaving radiance at 555 nm (LOWLW). The limiting of the high radiance flag to only the NIR bands accounted for only 1.8% of the gain in samples (from the HILT flag). Modification of the NIR iteration control also increased retrievals in some of the high-chlorophyll regions, primarily due to the new algorithm's ability to reset the starting chlorophyll at each iteration. While the change to the absorption coefficient in the NIR correction had the effect of lowering chlorophyll retrievals in the highest-chlorophyll waters, the chlorophyll was essentially unchanged in open-ocean regions. Globally, the effect of the miscellaneous changes on chlorophyll concentration is a decrease of less than about 1.5%.

Summary Statstics: Median Percent Change from Case 3
Filled Bins Samples/Bin chlor_a nLw(412) nLw(490) nLw(555) tau(865)
Monthly Daily Monthly Daily Monthly Daily Monthly Daily Monthly Daily Monthly Daily Monthly Daily
+0.8 +3.4 +3.0 +3.3 -1.35 -1.52 -0.09 -0.13 +0.61 +0.53 -0.28 -0.48 +0.81 +0.85
Scatter Plot of Chlorophyll - Case5:Case4

Case 5: Change in Near IR Correction Method

(ALGTEST 4)

This case incorporates the change from a chlorophyll -based to a reflectance-based NIR correction. The NIR correction as implemented in reprocessing #3 tended to depress the 510-865 Ångstrom values, often forcing the aerosol model to the oceanic model, which resulted in depressed chlorophyll retrievals.

The changes to the NIR correction will tend to force the aerosol model to the oceanic model less often, and thereby provide a more reasonable aerosol retrieval, consistent with aerosols retrieved over nearby waters where the NIR correction was not applied. In general, the water-leaving radiances are lowered slightly. The effect of this on the chlorophyll product is to increase the high chlorophyll retrievals. Some moderate chlorophylls are decreased. The impact of this change is primarily seen for chlorophyll concentrations greater than 1 mg m-3.

Summary Statstics: Median Percent Change from Case 4
Filled Bins Samples/Bin chlor_a nLw(412) nLw(490) nLw(555) tau(865)
Monthly Daily Monthly Daily Monthly Daily Monthly Daily Monthly Daily Monthly Daily Monthly Daily
+0.0 -0.1 -0.1 -0.1 +0.00 +0.00 +0.00 +0.00 +0.00 +0.00 +0.00 +0.00 +0.00 +0.00
Scatter Plot of Chlorophyll - Case6:Case5

Case 6: Change to Glint Mask Use

(ALGTEST 5)

This case incorporates the change to eliminate masking based on the high glint flag. At reprocessing #3, a glint correction was introduced for pixels where glint radiance was less than the threshold set to trigger the high glint flag. Wang and Bailey (2000) demonstrated that this correction was reasonable up to the level where the high Lt (TOA radiances above the knee) flag is triggered.

The effect is a reduction in the number of masked pixels in the glint region, which results in an increase in the coverage in the daily files and the number of samples over longer time scales. In the May 1999 monthly, the numbers of samples and bins increased by about 0.8% and 0.2%, respectively. The increase in samples/bins occurs only near the sub-solar point.

Summary Statstics: Median Percent Change from Case 5
Filled Bins Samples/Bin chlor_a nLw(412) nLw(490) nLw(555) tau(865)
Monthly Daily Monthly Daily Monthly Daily Monthly Daily Monthly Daily Monthly Daily Monthly Daily
+0.2 +0.9 +0.8 +1.0 +0.00 +0.00 +0.00 +0.00 +0.00 +0.00 +0.00 +0.00 +0.00 +0.00
Scatter Plot of Chlorophyll - Case7:Case6

Case 7: Improved Cloud Flagging Method

(ALGTEST 6)

This case incorporates the new cloud flagging algorithm based on Band 8 surface reflectance which better compensates for the increasing Rayleigh path radiance with solar and viewing angle. The reprocessing #3 algorithm, based on TOA reflectance, was found to be overly restrictive at higher solar and view angles.

The effect is an overall reduction in the number of pixels flagged for clouds, mainly at high solar zenith angles (i.e., extreme latitudes). This results in an increase in the number of filled bins and samples per bin in this region. There is also a small increase in the number of flagged pixels in the mid-latitudes. In the May 1999 monthly, the net increase in the samples and bins was about 1.5% and 1.3%, respectively.

Summary Statstics: Median Percent Change from Case 6
Filled Bins Samples/Bin chlor_a nLw(412) nLw(490) nLw(555) tau(865)
Monthly Daily Monthly Daily Monthly Daily Monthly Daily Monthly Daily Monthly Daily Monthly Daily
+1.4 +1.5 +1.5 +1.3 +0.00 +0.00 +0.00 +0.00 +0.00 +0.00 +0.00 +0.00 +0.00 +0.00
Scatter Plot of Chlorophyll - Case8:Case7

Case 8: Reduce Stray Light Mask

(ALGTEST 7)

This modification incorporates the change to the stray light masking algorithm, in which the number of straylight-masked pixels was reduced from 3 to 2 before and after a bright target. The original masking/correction scheme for stray light (GAC resolution), determined pre-launch, was conservative with the intention that the masking/correction scheme be revisited after launch. The change is only implemented for GAC resolution data. The masking for LAC resolution data is already optimal. Straylight corrections were applied to the additional unmasked pixels.

The effect is a reduction in the number of masked pixels, especially around clouds and land. This results in a significant increase in the number of Level-3 bins filled, and in the number of samples averaged into each bin. The increased coverage is most significant on daily time-scales, where the number of filled bins typically increased by 16%. The increase in filled bins for a typical 8-day composite was 9%. For the May 1999 monthly composite, the numbers of samples and bins increased by about 16% and 3%, respectively.

Summary Statstics: Median Percent Change from Case 7
Filled Bins Samples/Bin chlor_a nLw(412) nLw(490) nLw(555) tau(865)
Monthly Daily Monthly Daily Monthly Daily Monthly Daily Monthly Daily Monthly Daily Monthly Daily
+3.0 +16.0 +16.0 +16.0 +0.00 +0.00 +0.00 +0.00 +0.00 +0.00 +0.00 +0.00 +0.00 +0.00
Ratio Image of nLw(412)- Case 9:Case 8

Case 9: Add Fresnel Transmittance Correction, Chlorophyll Failure Condition Modification and Final Gains

(ALGTEST17)

This case modifies the normalization of water-leaving radiance to include a correction for Fresnel transmittance through the water-atmosphere interface. The effect will be to increase the nLw in all bands, with the largest increase occuring at the highest view zenith angle (3% at the GAC limit of 56-deg). In addition, a few other minor changes were introduced at this stage.

The number of match-up points used in the vicarious calibration was further reduced by the exclusion of 6 points with moderate glint contamination. The effect of removing these points is to very slightly raise the gains (all things being equal). Additionaly, four more lunar calibration points were added to the data set used to compute the time dependence correction for the calibration table.

Under certain geometric conditions, the T99 and C50 aerosol models cross-over in epsilon space, causing discontinuities when the aerosol path radiances are extrapolated into the visible. These discontinuities appear along lines of constant scattering angle, and they are sometimes visible in images of water-leaving radiance and even chlorophyll. The epsilon-targeted smoothing makes these effects even more apparent, as the aerosol-model-selection noise is reduced across the scattering-angle isolines. A fix has been developed which identifies these model cross-over conditions and revises the model selection result accordingly. This is a relatively rare problem which will not significantl effect global results.

The HDF type for Level-2 chlorophyll products was changed from scaled int16 to float32. The new format will eliminate the quantization issue seen at very low chlorophylls. In conjuction with this change, the CHLWARN flag was redfined to trip when chlor_a is outside of the range (0.0-100) mg/m^3, and the redefined flag is now applied as a mask at Level-3. The was neccessary because, with the old int16 scaling, any chlor_a above 64.767 was inherently truncated to 64.767. But with the new, unscaled chlor_a, we can store chlor_a up to the limits of the OC4 algorithm (0.0-640) mg/m^3, so we mask on the CHLWARN flag to avoid adding any extreme outliers to the binned average.

After posting the initial results, it was pointed out by Dr. Watson Gregg that the variance in the reprocessing #4 chlorophyll product was increased, particulary for high solar zenith angle data. Upon closer inspection, it was discovered that near cloud edges a number pixels with anomalously high chlorophyll values were being binned. The majority of these pixels had unrealistic radiance spectra. Two simple tests were implemented to mask pixels with these characteristics. The first requires that longer wavelenghts cannot go negative before shorter wavelengths and bands 4 and 5 must be positive for chlorhophyll to be calculated. The second test prevents chlorophyll from being calculated if reflectance in any band is negative by more than 0.1%. Implementing these tests eliminated the majority of the cloud edge speckles.

Summary Statstics: Median Percent Change from Case 8
Filled Bins Samples/Bin chlor_a nLw(412) nLw(490) nLw(555) tau(865)
Monthly Daily Monthly Daily Monthly Daily Monthly Daily Monthly Daily Monthly Daily Monthly Daily
+0.1 +0.4 +0.4 +0.5 +3.23 +3.21 -0.56 -0.56 -0.56 -0.56 +1.04 +1.07 -1.19 -1.13

Scatter Plot of Chlorophyll - Reprocessing #4: Reprocessing #3Cumulative Changes: Reprocessing #4 Baseline

Significant improvements over Reprocessing #3:

  • Improved radiance retrievals
  • Improved chlorophyll retrievals
  • Overall increase in global coverage
The cumulative effect of the changes made for reprocessing 4 increased the number of samples retrieved in a monthly by 27.76% and the % of bins in a monthly by over 6%. The bin coverage increases more for shorter time scales. For an 8-day period the increase is 13% and for a daily, the increase is about 24%. The increase was primarily due to the relaxation of the stray light masking criteria , accounting for 59.8% of the gain, but other contributions came from the improved treatment of conditions having very low aerosol radiances, the relaxed flagging for high total radiances and the improved cloud flagging algorithm.

Summary Statstics: Median Percent Change from Operational (Reprocessing #3)
Filled Bins Samples/Bin chlor_a nLw(412) nLw(490) nLw(555) tau(865)
Monthly Daily Monthly Daily Monthly Daily Monthly Daily Monthly Daily Monthly Daily Monthly Daily
+5.98 +24.7 +24.6 +24.3 -7.10 -7.59 +17.3 +16.6 +5.13 +4.70 +3.59 +2.84 -1.34 -3.38

Detailed Analysis