Statistical regularization for trend detection: an integrated approach for detecting long-term trends from sparse tropospheric ozone profiles

Detecting a tropospheric ozone trend from sparsely sampled ozonesonde profiles (typically once per week) is challenging due to the short-lived anomalies in the time series resulting from ozone's high temporal variability. To enhance trend detection, we have developed a sophisticated statistical approach that utilizes a geoadditive model to assess ozone variability across a time series of vertical profiles. Treating the profile time series as a set of individual time series on discrete pressure surfaces, a class of smoothing spline ANOVA (analysis of variance) models is used for the purpose of jointly modeling multiple correlated time series (on separate pressure surfaces) by their associated seasonal and interannual variabilities. This integrated fit method filters out the unstructured variation through a statistical regularization (i.e., a roughness penalty) by taking advantage of the additional correlated data points available on the pressure surfaces above and below the surface of interest. We have applied this technique to the trend analysis of the vertically correlated time series of tropospheric ozone observations from (1) IAGOS (In-service Aircraft for a Global Observing System) commercial aircraft profiles above Europe and China throughout 1994–2017 and (2) NOAA GML's (Global Monitoring Laboratory) ozonesonde records at Hilo, Hawaii, (1982–2018) and Trinidad Head, California (1998–2018). We illustrate the ability of this technique to detect a consistent trend estimate and its effectiveness in reducing the associated uncertainty in the profile data due to the low sampling frequency. We also conducted a sensitivity analysis of frequent IAGOS profiles above Europe (approximately 120 profiles per month) to determine how many profiles in a month are required for reliable long-term trend detection. When ignoring the vertical correlation, we found that a typical sampling strategy (i.e. four profiles per month) might result in 7 % of sampled trends falling outside the 2σ uncertainty interval derived from the full dataset with an associated 10 % of mean absolute percentage error. Based on a series of sensitivity studies, we determined optimal sampling frequencies for (1) basic trend detection and (2) accurate quantification of the trend. When applying the integrated fit method, we find that a typical sampling frequency of four profiles per month is adequate for basic trend detection; however, accurate quantification of the trend requires 14 profiles per month. Accurate trend quantification can be achieved with only 10 profiles per month if a regular sampling frequency is applied. In contrast, the standard separated fit method, which ignores the vertical correlation between pressure surfaces, requires 8 profiles per month for basic trend detection and 18 profiles per month for accurate trend quantification. While our method improves trend detection from sparse datasets, the key to substantially reducing the uncertainty is to increase the sampling frequency.

Statistical regularization for trend detection: An integrated approach for detecting long-term trends from sparse tropospheric ozone profiles

Detecting a tropospheric ozone trend from sparsely sampled ozonesonde profiles (typically once per week) is challenging due to the noise in the time series resulting from ozone's high temporal variability. To enhance trend detection we have developed a sophisticated statistical approach that utilizes a geoadditive model to assess ozone variability across a time series of vertical profiles. Treating the profile time series as a set of individual time series on discrete pressure surfaces, a class of smoothing spline ANOVA (analysis of variance) models is used for the purpose of jointly modeling multiple correlated time series (on separate pressure surfaces) by their associated seasonal and interannual variabilities. This integrated fit method filters out the unstructured noise through a statistical regularization (i.e. a roughness penalty), by taking advantage of the additional correlated data points available on the pressure surfaces above and below the surface of interest. We have applied this technique to the trend analysis of the vertically correlated time series of tropospheric ozone observations from 1) IAGOS (In-service Aircraft for a Global Observing System) commercial aircraft profiles above Europe and China, and 2) NOAA GMD's (Global Monitoring Division) ozonesonde records at Hilo, Hawaii and Trinidad Head, California. We illustrate the ability of this technique to detect a consistent trend estimate, and its effectiveness for reducing the associated uncertainty in the noisy profile data due to low sampling frequency. We also conducted a sensitivity analysis of frequent IAGOS profiles above Europe (approximately 120 profiles per month) to determine how many profiles in a month are required for reliable long-term trend detection. When ignoring the vertical correlation we found that a typical sampling strategy of 4 profiles-per-month results in 7 % of sampled trends falling outside the 2-sigma uncertainty interval derived from the full data set, with associated 10 % of mean absolute percentage error. We determined that an optimal sampling frequency is 14 profiles per month when using the integrated fit method for calculating trends; when the integrated fit method is not applied, the sampling frequency had to be increased to 18 profiles per month to achieve the same result. While our method improves trend detection from sparse data sets, the key to substantially reducing the uncertainty is to increase the sampling frequency.