NOTE: to be updated based on recent software delivery (January 24, 2011)

Background:

The philosophy of the NASA/GSFC Ocean Data Processing Group has always involved the use of rapid prototypes evolving into ever more realistic full-mission simulations in order to verify that all requirements are being met, that all components of the system are being thoroughly exercised and that through the use of "real-world" testing of the end-to-end system on a routine basis, that REAL problems will be encountered and remedial actions can be developed and tested. We have seen too many projects adopt the "test for success" strategy with limited performance tests conducted under ideal conditions or to focus primarily on resolving anticipated and well orchestrated anomalies. While it is good to anticipate and plan for known contingencies, we have found that it is either the lack of continuous full-system tests for as long before launch as possible or the unplanned events that always seem to occur at the most inconvenient times that create the most difficulties. The three guiding principles that we have found invaluable to follow are:

1- Maintain Flexibility - things WILL change
2- Strive for maintainability of code and documentation
3- Testing, Testing and more Testing: Get the mechanics right so that when data starts to flow, you can focus on the science

Our approach has been to start with a full end-to-end data flow description and then to break it down into its component parts and where possible, use as close to real data and operational software as possible and more importantly, to continuously evolve the simulation to add more credible data or software as they either become available or are developed. In this way, all the supporting infrastructure, tools and procedures can be developed, tested and refined as the process continues. If done in a modular, flexible way, the entire system can gradually evolve to a higher and higher level of fidelity until you have as close to a fully tested and functional launch-ready system as possible prior to launch.

The basic data flows involved in the current simulation which will evolve into the operational data flow/system are as follows:

1) An Aquarius science downlink file, containing multiple copies of the contents of the Aquarius instrument's memory, along with the SAC-D telemetry and ephemeris files appear on the CONAE data server a short time after each scheduled X-band transmission.
2) The telemetry, ephemeris, and downlink files are detected by the site-scanner component of the Ingest Subsystem that runs within the Aquarius Data Processing System (ADPS) and queues the newly discovered files for ingest.
3) The ingest process retrieves these files and schedules the ephemeris files for processing to produce orbit-time information and the downlink files for processing to Level-0.
4) Level-0 process removes duplicate records, sorts the file by time and creates a metadata file describing the contents of the Level-0 file to be used by the web-based downlink monitor.
5) Level-1A process unpacks the Aquarius science and telemetry data from the science blocks, writes these to temporary Level-1A files for each orbit segment contained in the Level-0 file, and adds time-appropriate portions extracted from the SAC-D telemetry and ephemeris files, along with a metadata file describing the contents of the Level-1A file, to be used by the web-based downlink monitor.
6) The Telemetry Analysis process reads each Level-1A file as it is produced, analyzes the Aquarius housekeeping telemetry, and creates an html-formatted report file which is placed in a web-accessible directory and linked immediately to the web-based downlink monitor.
7) Level-1A merge process uses the orbit-time information produced by the ephemeris files to combine all incomplete and duplicate orbit segments into a single complete Level-1A file per orbit, with 10 minutes of overlap required for the Level-1B process.
8) Level-2 process, using all required ancillary data that has been pre-ingested and preprocessed as required, applies the sensor calibration to the data and creates derived geophysical parameter files as well as browse files for the web-based browse and order tool.
9) Insitu data locations and times received from the Aquarius Validation and Data System (AVDS) are checked against newly created Level-2 files and where "matches" are found, a Level-2 extracted file is created and made available to the AVDS for matchup analysis which results in updated to the processing coefficients to improve the Aquarius-derived salinity retrievals. The matchup process is described here.
10) Level-3 binning process creates time averaged global fields (daily, weekly, monthly, etc.) without any data smoothing/filtering to asses the quality of the observations just as they were observed.
11) Level-3 smoothing process uses the methodology outlined here to produce the final Level-3 binned products.
12) The Level-3 binned files are mapped to output any desired parameter and time resolution on a global grid at 1 degree resolution along with browse files for the web-based browse and order tool.

Background documentation on the development of the simulated Aquarius data set that has been used to develop and test the data flows and access to the sample data products can be found here.

The Mission Simulation starts with simulated sensor data generated by members of the Aquarius Science Team, using assumed salinity fields and spacecraft orbit data, actual ancillary data and models of the instrument response (Part 1 in the diagram above).

Once the simulated sensor files are created, these are formatted into Aquaris science blocks and packaged into realistic downlink files based on predicted contact times and durations which are as close to the actual downlink data format as possible given the currently available information and sample telemetry (Part 2 in the diagram below). Actual Aquarius instrument telemetry (radiometer, scatterometer and instrument subsystem), acquired during the instrument integration and test phase recently completed at JPL, are used to populate the appropriate telemetry fields in the simulated downlink files (repeated as needed to fill the complete orbit) although it must be stated that there is no realistic relationship between the telemetry values from instrument I&T and the simulated science data. However, the telemetry values will be useful in testing the telemetry analysis tools and web displays since they represent realistic telemetry values. It is important to state that there is no feasible way to have consistency between realistic instrument telemetry values and the corresponding science data from which sea surface salinity and surface roughness can be derived. After the Aquarius instrument is integrated onto the SAC-D observatory and testing begins, we will verify and if necessary, update the downlink format.

For the current simulated mission test (a year long data set for 2007 is currently being used), approximately 4 or 5 downlink files per day were created containing the appropriate duplicate records based on the predicted downlink duration and times. Using the same predicted downlink schedule, a new downlink file was placed in the CONAE data server directory along with the attitude and ephemeris files. The site scanner process detects these new files as described above, and the process begins as if CONAE had acquired an actual Aquarius downlink.

The data flow outlined below provides an overview of the major steps in Aquarius data processing as it has currently been implemented within the Ocean Data Processing System (ODPS) and shows the top-level view of the level conversion steps as well as the interaction with the web-based telemetry monitoring tool. What is not shown in this diagram is the nearly real-time availability of all data products via both our ftp and web-based browse/order/subscription system.That component currently exists for all of the other missions supported by the ODPS and is currently being upgraded to include support for Aquarius.

The diagrams and code description below outline the details and logic for each of the major steps in the Aquarius data processing flow as it currently exists and makes note of the work that remains to be done.

L0GEN_AQUARIUS (Removes duplicated L0 records and sorts by time)

NOTE: This routine now sorts on last "last block number" field within the ICDS Processing Status housekeeping (IP_LST_BLK_NUM) rather than the "GPS time tag/time tag offset" fields of the record header. This provides a more reliable record order than the time which is vulnerable to occasional conversion errors.

L1AGEN_AQUARIUS (distributes L0 entries into temporary L1 orbit files)

NOTE: The output block number in the L1A files is now based on the input L0 ICDS block number mentioned above.

L1AMERGE_AQUARIUS (Merges temp L1 files into single L1 orbit file)

QMET_ATM_AQUARIUS

1) Calls "gen_qmet_atm.csh" for current day.

1) For each 6hr QMET file (00 06 12 18) ...
- 1) Check that QMET filetime is at least 1 hour before current time.
- 2) Generate NCEP download script from "get_dss.csh.template"
- 3) Execute "get_dss.csh" to download NCEP file (fnl_YYYYMMDD_HH_00).
- 4) Convert from grib to grib2 format with "togrib2.csh"
- 5) Execute "runaquarius_met.sh" to generate:
  - Nyyyydoyhh_QMET_NCEP_6h
- geohgt Geopotential height at the surface (gpm)
- tmp_2m Temperature 2m above surface (K)
- press Surface pressure (Pa)
- rh Relative humidity 2m above surface (%)
- p_water Precipitable water:entire atmosphere (kg m-2)
- c_water Cloud water:entire atmosphere (kg m-2)
- u_wind u-component of wind 10m above surface (m s-1)
- v_wind v-component of wind 10m above surface(m s-1)
- geohgt_prfl Geopotential height profile (gpm)
- tmp_prfl Temperature profile (K)
- rh_prfl Relative humidity profile (%)
- clwmr_prfl Cloud water mixing ratio (kg kg-1)
  - at the following levels:
  - Isobaric surface: 1000 mbar
  - Isobaric surface: 975 mbar
  - Isobaric surface: 950 mbar
  - Isobaric surface: 925 mbar
  - Isobaric surface: 900 mbar
  - Isobaric surface: 850 mbar
  - Isobaric surface: 750 mbar
  - Isobaric surface: 700 mbar
  - Isobaric surface: 650 mbar
  - Isobaric surface: 600 mbar
  - Isobaric surface: 550 mbar
  - Isobaric surface: 500 mbar
  - Isobaric surface: 450 mbar
  - Isobaric surface: 400 mbar
  - Isobaric surface: 350 mbar
  - Isobaric surface: 300 mbar
  - Isobaric surface: 250 mbar
  - Isobaric surface: 200 mbar
  - Isobaric surface: 150 mbar
  - Isobaric surface: 100 mbar
  - Isobaric surface: 70 mbar
  - Isobaric surface: 50 mbar
  - Isobaric surface: 30 mbar
  - Isobaric surface: 20 mbar
  - Isobaric surface: 10 mbar
- 6) Compress with bzip2
- 7) Move to ancillary directory
- 8) Execute "mk_atm_tb_aquarius.csh" to generate:
  - Nyyyydoyhh_QATM_NCEP_6h.h5 Absorption_CLD Absorption_H2O Absorption_O2 Downwelling_TB Transmittance Upwelling_TB
  - - Takes ~20 minutes to run
- 9) Compress with bzip2 and move to ancillary directory

L2GEN_AQUARIUS

L2BIN_AQUARIUS

L3BIN_AQUARIUS

L2SMOOTH_AQUARIUS

SMIGEN