PACE Science Data Reprocessing: Version 2
The initial public release of PACE science data products (Version 1) began on 11 April 2024, and provided the science and applications user community with access to the Level-1 data and a limited suite of derived products from the OCI, HARP2, and SPEXone instruments, with the caveat that the data were in a highly preliminary state and should be used with caution. Reprocessing Version 2 is the first full mission reprocessing, and primarily serves to incorporate improved calibration knowledge from on-orbit measurements collected by the three PACE instruments. The reprocessing will also include all standard science mode data collected during the PACE commissioning period, starting from instrument engineering first light in February 2024.
As in Version 1, all data products are distributed in self-describing netCDF4 format and are available in the following data processing levels.
- Level-1A, raw instrument data and spacecraft telemetry, reformatted to netCDF4
- Level-1B, calibrated & geolocated instrument data
- Level-1C, calibrated, geolocated, and co-registered to a common grid
- Level-2, derived geophysical science data products
- Level-3, temporally and spatially composited (binned and mapped) global products
- Level-4, geophysical products derived from combined Level-3 inputs and/or models
The Level-1C and Level-2 data files are divided into 5-minute granules that start at the southern terminator crossing of the ascending daylight orbit and end at the northern terminator crossing. For OCI, and HARP2 the Level-1A and Level-1B data are also in 5-minute granules, while SPEXone is organized into continuous daylight orbit periods (~50 minutes).
The products available at this release are categorized into maturity levels as:
- Provisional: results have been reviewed and are in family with heritage data products or other basis of expectation, but which have not yet been validated and may still contain significant errors.
- Test: results have not yet been reviewed by algorithm developers and or may be known to have substantial errors in implementation that are under investigation.
- Diagnostic: products that are produced to support analysis of algorithm behavior, but that are not intended for science.
The notes that follow provide an overview of the changes introduced between Version 1 and Version 2, for the Level-1 products of each instrument as well as the derived science products currently available. For a comprehensive list of the science data products planned for PACE and the current status of those products and algorithms see the PACE Data Products Table.
OCI Level-1A/B/C Data (Provisional)
Changes from Version 1:
- The OCI absolute calibration coefficients were revised by combining on-orbit solar diffuser measurements and prelaunch GLAMR measurements. The solar diffuser based calibration coefficients below 900nm are adjusted to agree with the longer range spectral trends of the GLAMR coefficients, eliminating noise that was apparent in the GLAMR coefficients regarding mirror side ratios and adjacent spectral channels. Calibration coefficients of the SWIR channels are purely based on solar diffuser measurements, because a hysteresis correction has not yet been applied to the prelaunch data. The SWIR band calibration will be updated in a future reprocessing.
- Geolocation accuracy was significantly improved and is now generally within 1 pixel at all scan angles.
- The bug in SWIR band L1B processing that caused invalid values in the first scan line of each file was resolved.
Known Issues and Data Characteristics:
- The optical design of the OCI SWIR detector assembly (SDA) causes the bands to view different locations along-scan at a given time, and the data are packetized by time. The bands are pixel-shifted into alignment with the hyperspectral bands. This results in fill pixels at the start or end of the scan, depending on the required pixel shift. The worst case is the 2260 nm band that has 13 fill pixels at the start of the scan.
- Data below 340nm has not been characterized prelaunch as completely and accurately as the data above 340nm. The data below 340nm is only released to facilitate assessment and potential refinement of radiometric accuracy and should not be considered as science quality.
- Data in the transition region between the red and blue focal planes, between 590 and 610nm, shows significant discontinuity. The discrepancy varies with mirror side and possibly with scan angle. Measurements in this range have much higher uncertainty and should not be used for science algorithms.
- Some SWIR bands show significant signs of apparent band-to-band registration issues. SWIR data (especially bands 1038nm, 1250nm Standard Gain, and 1615nm Standard Gain) with large radiance gradients to neighboring pixels in scan direction.
- The temporal gain trend of the calibration in this update is derived from measurements of the daily solar diffuser (SD) measurements, assuming that the SD reflectance is stable. In a future update, the monthly solar diffuser measurements will be used to reduce the impact of changes in solar diffuser reflectance on the trend in measured top-of-atmosphere radiances. The impact will likely be small above 450nm (less than 0.2%), but larger at shorter wavelengths.
- Occasional noise spikes are seen in OCI dark data, especially in the area of the Southern Atlantic Anomaly. These spikes corrupt the background subtraction for a complete scan line, leading to striping in the L1B data. The algorithm to calculate the background subtraction in the L1B code will be improved to remove these noise spikes in the next reprocessing. It is likely that similar noise spikes are present in individual pixels of the earth view data as well. Handling earth view noise spikes will be a longer term effort.
- Further improvements in geolocation accuracy are on-going. It has been determined that the PACE spacecraft on-board ephemerous data currently used for geolocation of OCI produces uncertainties of order 150 meters. This will be resolved in refined processing through use of definitive ephemerous data derived in ground-based post processing of the GPS data.
HARP2 Level-1A/B/C Data (Provisional)
On-orbit calibration data has been collected during solar and lunar calibration exercises, and a refined calibration is included in the version 2 release.
Changes from Version 1:
- Improvements to instrument calibration based on data collected during on-orbit solar and lunar calibration exercises. This includes:
- Correction for elevated brightness observed in V1
- Refined, FOV-dependent radiometric & polarimetric characterization
- Updated dark offsets
- V2 results are now within +/-3% in comparison with OCI (subject to verification, one-sigma)
- Geolocation accuracy now within one Level-1B pixel for all angles through:
- Improved optical model (spectral-dependent magnification & distortion, realistic MTDI sampling)
- Improved HARP2 boresight offsets
- Improved detector alignment for reducing false polarization
Known issues:
- Polarimetric performance is subject to comprehensive evaluation, particularly in the accuracy of DoLP and reference plane orientation.
- Blue band exhibits both positive and negative bias relative to other bands when compared with SPEXOne over multiple orbits spanning several days. This bias is dependent on the viewing angle and latitude. Efforts are currently underway to investigate the origins of this bias and to develop appropriate corrections.
- The red scale factor trends higher than the non-red factor by about 25%. This will result in a larger subtraction of the red data as compared to the non-red data. This will affect the radiometric data and we need to test the new correction.
- Geolocation performance is subject to more comprehensive evaluation.
- Alignment related false polarization needs further evaluation and improvement
SPEXone Level-1A/B/C Data (Provisional)
Changes from Version 1:
- Geolocation accuracy has been improved.
- Additional bad detector pixels have been identified and masked.
- Unphysical oscillations in the degree of polarization (DoLP) and radiance have been reduced by applying a small shift to the detector image.
- Radiometric response accuracy has been improved.
Known issues:
- Reduced DoLP accuracy at the shortest wavelengths.
OCI Level-2 Products
An expanded set of OCI Level-2 science data products is being released at this time. They are organized into product suites, with each suite in one file per granule. In some cases, the Level-2 products suites contain a mix of Provisional, Test, and Diagnostic products.
OC_AOP, Ocean Color Apparent Optical Properties:
Provisional Products:
- Remote Sensing Reflectance (Rrs)
- Rrs Uncertainties (Rrs_unc)
Test Products:
- Apparent Visible Wavelength (avw)
Diagnostic Products:
- Aerosol Optical Thickness (aot)
- Aerosol Angstrom Exponent (angstrom)
Changes from Version 1.0:
- The updated absolute calibration of OCI changed the magnitude of Rrs retrievals in all wavelengths, and resulted in a significant reduction in suspect band-to-band spectral variability. Mirror-side striping artifacts are also substantially reduced.
- An updated atmospheric gas transmittance correction that now includes effects of O2-O2 collision-induced absorption resulted in reduced residual gas transmittance artifacts in the Rrs retrievals (e.g., near 360, 380, 477, 577, and 630nm).
- Data are now flagged as reduced quality for view zenith angles greater than 60-deg. These flagged data are masked in Level-3 products. In V1 the data was flagged at 75-deg, and thus extended to the edge of the OCI swath. This results in larger gaps in L3 products between orbits, but removed highly suspect Rrs retrievals along the swath edge from Level-3 Rrs products and all products derived from Rrs
- Data within 1 pixel along track and within 2 pixels across track from a bright source such as a cloud are flagged as reduced quality, based on pre-launch characterization of bright source recovery (e.g., straylight). These straylight flagged pixels are masked in Level-3 products. No straylight flagging was done in V1. Result is increased masking around clouds.
Known Issues:
- No vicarious calibration has yet been applied. It is expected that a vicarious calibration will be applied in the next reprocessing, to further reduce bias in the Rrs retrievals relative to ground truth.
- Limited validation of Rrs retrievals has been performed. Results show good agreement (to first order) with ground truth and with Rrs retrievals from heritage multispectral satellite missions (e.g., VIIRS), but more sampling is needed to fully assess data quality.
- Performance of Rrs retrieval in the UV (below 400nm) has not yet been meaningfully reviewed and is likely to contain significant biases and erroneous variability.
- Corrections for absorbing gases have been applied, but refinement is on-going. Rrs variability, especially in the red, is likely to contain residual artifacts from water vapor and oxygen absorption near 680nm, 720nm, 760nm, and 820nm.
- As noted above, there is a discontinuity in the observed radiances at the transition between the blue and red focal planes. This results in an artifact in Rrs in the 590-610nm region. Science algorithms should avoid use of data in this region.
- The current processing extends to higher view zenith angles than the heritage sensors. The atmospheric correction becomes increasingly difficult at these extreme geometries, and erroneously elevated reflectance has been observed in red wavelengths near scan edge. These data are flagged at Level-2 and masked at Level-3.
- Chlorophyll Fluorescence Line Height will be included in a future release.
OC_IOP (Ocean Inherent Optical Properties):
Provisional Products:
- Total absorption coefficient (a)
- Total backscatter coefficient (bb)
- Phytoplankton absorption coefficient (aph)
- Phytoplankton absorption coefficient standard uncertainty (aph_unc)
- Non-algal particle plus chromophoric dissolved organic matter absorption coefficient (adg)
- Non-algal particle plus chromophoric dissolved organic matter absorption coefficient standard uncertainty (adg_unc)
- Particle backscattering coefficient (bbp)
- Particle backscattering coefficient standard uncertainty (bbp_unc)
Test Products:
Diagnostic Products:
- Mean residual of model fit (rrsdiff)
Changes from Version 1.0:
- Products are derived from Rrs. See OC_AOP changes.
Known issues:
- Products are derived from Rrs. See OC_AOP: Rrs known issues.
- Some algorithm failure and artifacts noted in highly productive and near-shore waters. Future refinement will improve these missing data.
OC_BGC (Ocean Biogeochmical Properties):
Provisional Products
- Phytoplankton Chlorophyll-a (chlor_a)
- Phytoplankton Chlorophyll-a uncertainties (chlor_a_unc)
Test Products
- Phytoplankton Carbon (carbon_phyto)
- Phytoplankton Carbon uncertainties (carbon_phyto_unc)
- Particulate Organic Carbon (poc)
Changes from Version 1.0:
- Products are derived from Rrs. See OC_AOP changes.
Known issues:
- Products are derived from Rrs. See OC_AOP: Rrs known issues.
- Circular artifacts observed near subsolar point, where OCI is tilting through nadir and glint is extreme. Flagged as high glint in Level-2. Masked in Level-3.
- Particulate Inorganic Carbon (pic) and poc_unc will be included in a future release.
PAR (Photosynthetically Available Radiation)
Test Products
- Daily Mean PAR, Below-surface, Scalar (par_day_scalar_below)
- Daily Mean PAR, Above-surface, Planar (par_day_planar_above)
- Daily Mean PAR, Below-surface, Planar (par_day_planar_below)
- Instantaneous PAR, Below-surface, Planar (ipar_planar_above)
- Instantaneous PAR, Below-surface, Planar (ipar_planar_below)
Changes from Version 1.0:
Known issues:
CLDMASK (Cloud Mask)
Test Products
- Cloud Mask (cloud_flag)
- Cloud and Cloud-adjacent Mask (cloud_flag_dilated)
Changes from Version 1.0:
- This is the first public release for these products.
Known issues:
- The current implementation of the MERRA2 snow/sea ice mask can cause blockiness around coasts where the land is snow-covered. This will be fixed in a forthcoming release.
CLD (Cloud Optical Properties and Cloud Altitude)
Test Products
- Cloud Top Pressure (ctp)
- Cloud Top Temperature (ctt)
- Cloud Top Height (cth)
- Cloud Effective Radius using 2.2 µm band (cer_22)
- Cloud Optical Thickness using 2.2 µm band (cot_22
- Cloud Water Path using 2.2 µm band (cwp_22)
- Cloud Effective Radius using 2.1µm band (cer_21)
- Cloud Optical Thickness using 2.1µm band (cot_21
- Cloud Water Path using 2.1um band (cwp_21)
- Cloud Effective Radius using 1.6 µm band (cer_16)
- Cloud Optical Thickness using 1.6 µm band (cot_16)
- Cloud Water Path using 1.6 µm band (cwp_16)
- Cloud Effective Radius using 1.6 µm and 2.1 µm bands (cer_16_21)
- Cloud Optical Thickness using 1.6 µm and 2.1 µm bands (cot_16_21)
- Cloud Water Path using 1.6 µm and 2.1 µm bands (cwp_16_21)
- Cloud Phase using 2.1 µm band (cld_phase_21)
- Cloud Phase using 2.2 µm band (cld_phase_22)
- Cloud Phase using 1.6 µm band (cld_phase_16)
- Cloud Phase using 1.6 µm and 2.1 µm bands (cld_phase_16_21)
Changes from Version 1.0:
- This is the first public release for these products.
SFREFL (Surface Reflectance)
Test Products
- Spectral Surface Reflectance (rhos)
Notes
- Data is cloud masked using the Cloud and Cloud-Adjacent CLDMASK product, which will mask extra pixels in the vicinity of clouds and bright targets.
Changes from Version 1.0:
- This is the first public release for these products.
Known issues:
- Data is limited to 50 spectral wavelengths between 339nm to 2260nm.
- Contribution from aerosols is not removed.
- Residual artifacts in the spectral vicinity of strong atmospheric gas absorption (e.g., water vapor and oxygen) is expected.
LAND (Land Surface Indices)
Test Products
- Enhanced Vegetation Index (evi)
- Normalized Difference Vegetation Index (ndvi)
- Normalized Difference Water Index (ndwi)
- Normalized Difference Infrared Index (ndii)
- Normalized Difference Snow Index (ndsi)
- Chlorophyll-Carotenoid Index (cci)
- Carotenoid Content Index (car)
- Chlorophyll Index Red Edge (cire)
- Photochemical Reflectance Index (pri)
- Modified Anthocyanin Reflectance Index (mari)
Notes
- Multi-band indices (NDVI, EVI, NDWI, NDII, CCI, NDSI) are calculated using aggregates of OCI spectral reflectances corresponding to relevant MODIS bandwidths.
- Data is cloud masked using the Cloud and Cloud-Adjacent CLDMASK product, which will mask extra pixels in the vicinity of clouds and bright targets.
Changes from Version 1.0:
- This is the first public release for these products.
Known Issues:
- The NDSI algorithm does not currently incorporate a snow impossible mask.
- Contribution from aerosols is not currently removed from surface reflectance data, which may cause residual artifacts in vegetation index calculations.
