File Structure at Three Processing Levels for the Ocean Color Instrument (OCI)¶

Authors: Anna Windle (NASA, SSAI), Ian Carroll (NASA, UMBC), Carina Poulin (NASA, SSAI)

PREREQUISITES

This notebook has the following prerequisites:

1. OCI Data Access

Summary¶

In this example we will use the earthaccess package to access an OCI Level-1B, Level-2, and Level-3 NetCDF file and open them using xarray.

NetCDF (Network Common Data Format) is a binary file format for storing multidimensional scientific data (variables). It is optimized for array-oriented data access and support a machine-independent format for representing scientific data. Files ending in .nc are NetCDF files.

XArray is a package that supports the use of multi-dimensional arrays in Python. It is widely used to handle Earth observation data, which often involves multiple dimensions — for instance, longitude, latitude, time, and channels/bands.

Learning Objectives¶

At the end of this notebook you will know:

1. How to find groups in a NetCDF file
2. How to use xarray to open OCI data
3. What key variables are present in the groups within OCI L1B, L2, and L3 files

Contents¶

1. Setup
2. Inspecting OCI L1B File Structure
3. Inspecting OCI L2 File Structure
4. Inspecting OCI L3 File Structure

1. Setup¶

We begin by importing all of the packages used in this notebook. If you have created an environment following the guidance provided with this tutorial, then the imports will be successful.

Set (and persist to your user profile on the host, if needed) your Earthdata Login credentials.

Back to top

2. Inspecting OCI L1B File Structure¶

Let's use xarray to open up a OCI L1B NetCDF file using earthaccess. We will use the same search method used in OCI Data Access. Note that L1B files do not include cloud coverage metadata, so we cannot use that filter.

We want to confirm we are running code on a remote host with direct access to the NASA Earthdata Cloud. The next cell has no effect if we are, and otherwise raises an error. If there's an error, consider the substitution explained in the OCI Data Access notebook.

Let's open the first file of the L1B files list:

Notice that this xarray.Dataset has nothing but "Attributes". We cannot use xarray to open multi-group hierarchies or list groups within a NetCDF file, but it can open a specific group if you know its path. The xarray-datatree package is going to be merged into xarray in the not too distant future, which will allow xarray to open the entire hieerarchy. In the meantime, we can use a lower level reader to see the top-level groups.

TODO: Link to a PACE/OCI user's guide, referencing the section which explains each group.

Let's open the "observation_data" group, which contains the core science variables.

Now you can view the Dimensions, Coordinates, and Variables of this group. To show/hide attributes, press the paper icon on the right hand side of each variable. To show/hide data reprensetaton, press the cylinder icon. For instance, you could check the attributes on "rhot_blue" to see that this variable is the "Top of Atmosphere Blue Band Reflectance".

The dimensions of the "rhot_blue" variable are ("blue_bands", "number_of_scans", "ccd_pixels"), and it has shape (119, 1709, 1272). The sizes attribute of a variable gives us that information as a dictionary.

Let's plot the reflectance at postion 100 in the "blue_bands" dimension.

Back to top

3. Inspecting OCI L2 File Structure¶

OCI L2 files include retrievals of geophysical variables, such as Apparent Optical Properties (AOP), for each L1 swath. We'll use the same earthaccess search for L2 AOP data. Although now we can use cloud_cover too.

Let's look at the "geophysical_data" group, which is a new group generated by the level 2 processing.

The Rrs variable has length 184 in the wavelength dimension, so the blue, red, and SWIR wavelengths have been combined. Let's map the Rrs at "wavelength_3d" position 100.

Right now, the scene is being plotted using number_of_lines and pixels_per_line as "x" and "y", respectively. We need to add latitude and longitude values to create a true map. To do this, we will create a merged xarray.Dataset that pulls in information from the "navigation_data" group.

Although we now have coordinates, they won't immediately help because the data are not gridded by latitude and longitude. The Level 2 data cover the original instrument swath and have not been resampled to a regular grid. Therefore latitude and longitude are known, but cannot be used immediately to "look-up" values like you can along an array's dimensions.

Let's make a scatter plot of the pixel locations so we can see the irregular spacing. By selecting a slice with a step size larger than one, we get a subset of the locations for better visualization.

Let's plot this new xarray.Dataset the same way as before, but add latitude and longitude.

Now you can project the data onto a grid. If you wanna get fancy, add a coastline.

Let's plot the full "Rrs" spectrum for individual pixels. A visualization with all the pixels wouldn't be useful, but limiting to a bounding box gives a simple way to subset pixels. Note that, since we still don't have gridded data (i.e. our latitude and longitude coordinates are two-dimensional), we can't slice on a built-in index. Without getting into anything complex, we can just box it out.

The line plotting method will only draw a line plot for 1D data, which we can get by stacking our two spatial dimensions and choosing to show the new "pixel dimension" as different colors.

We will go over how to plot Rrs spectra with accurate wavelength values on the x-axis in an upcoming notebook.

Back to top

4. Inspecting OCI L3 File Structure¶

At Level-3 there are binned (B) and mapped (M) products available for OCI. The L3M remote sensing reflectance (Rrs) files contain global maps of Rrs. We'll use the same earthaccess method to find the data.

OCI L3 data do not have any groups, so we can open the dataset without the group argument. Let's take a look at the first file.

Notice that OCI L3M data has lat and lon coordinates, so it's easy to slice out a bounding box and map the "Rrs_442" variable.

Also becuase the L3M variables have lat and lon coordinates, it's possible to stack multiple granules along a new dimension that corresponds to time. Instead of xr.open_dataset, we use xr.open_mfdataset to create a single xarray.Dataset (the "mf" in open_mfdataset stands for multiple files) from an array of paths.

We also use a new search filter available in earthaccess.search_data: the granule_name argument accepts strings with the "*" wildcard. We need this to distinguish daily ("DAY") from eight-day ("8D") composites, as well as to get the 0.1 degree resolution projections.

The paths list is sorted temporally by default, which means the shape of the paths array specifies the way we need to tile the files together into larger arrays. We specify combine="nested" to combine the files according to the shape of the array of files (or file-like objects), even though paths is not a "nested" list in this case. The concat_dim="date" argument generates a new dimension in the combined dataset, because "date" is not an existing dimension in the individual files.

A common reason to generate a single dataset from multiple, daily images is to create a composite. Compare the map from a single day ...

... to a map of average values, skipping "NaN" values that result from clouds.