Format Change - HDF4 to netCDF4

The Level 2 and Level 3 products produced by the Ocean Biology Processing Group will change file formats from HDF4 to netCDF4 with the multi-mission reprocessing planned for the summer of 2014. There are several reasons for making this significant change to the data products.

  • With the advent of HDF5, the current HDF4 format, while still funded, is no longer under active development.
  • Data interoperability improves.
  • The application of accepted international conventions improves metadata consistency.

why netCDF4?

  • netCDF4 was NASA-funded effort to improve:
    • interoperability among scientific data representations
    • integration of observations and model outputs
    • I/O for high-performance computing.
  • netCDF4 combines the netCDF and HDF5 data models, taking the desirable characteristics of each, while taking advantage of their separate strengths:
    • netCDF is popular and easy to use, and includes many tools and multiple implementations.
    • HDF5 is powerful, has high-performance, is efficient for storage and extensible.
  • The goal of netCDF4 is to make netCDF more suitable for high-performance computing and large datasets, and to provide a simple high-level application programming interface (API) for HDF
  • ESA has chosen netCDF4 as the format for the sensors on upcoming Sentinel-3 mission (the ENVISAT follow-on)
  • Efforts are underway to recognize netCDF and the Climate and Forecasting (CF) Conventions as international standards
  • Backward compatible with the previous versions of netCDF will be maintained for future versions
  • netCDF is a recognized data format for a large number of data visualization and manipulation packages (e.g. GIS packages, ENVI, IDL, Matlab, GrADS, Panoply), so using it could facilitate opening up our data sets to a wider community.

Metadata Changes

With the switch to netCDF there is a corresponding update to the product metadata. Our updated schema follows these standards and conventions:

ISO19115
Defines the schema required for describing geographic information and services. It provides information about the identification, the extent, the quality, the spatial and temporal schema, spatial reference, and distribution of digital geographic data.
Climate and Forecast (CF)
The netCDF-CF conventions are a set of codified recommendations for practices built around published specifications. While CF is a convention rather than an established metadata standard, CF is a critically important step towards better interoperability.
Attribute Convention for Data Discovery (ACDD)
The netCDF Group at Unidata has recommended netCDF attributes for data discovery. Many of these attributes correspond to general discovery metadata content, so they are available in many metadata standards.

NASA's Earth Science Data and Information System Standards Office provides a summary of standards and conventions that they approve.