Nonparametric Limits of Agreement in Method Comparison Studies: A Simulation Study on Extreme Quantile Estimation

Bland–Altman limits of agreement and the underlying plot are a well-established means in method comparison studies on quantitative outcomes. Normally distributed paired differences, a constant bias, and variance homogeneity across the measurement range are implicit assumptions to this end. Whenever these assumptions are not fully met and cannot be remedied by an appropriate transformation of the data or the application of a regression approach, the 2.5% and 97.5% quantiles of the differences have to be estimated nonparametrically. Earlier, a simple Sample Quantile (SQ) estimator (a weighted average of the observations closest to the target quantile), the Harrell–Davis estimator (HD), and estimators of the Sfakianakis–Verginis type (SV) outperformed 10 other quantile estimators in terms of mean coverage for the next observation in a simulation study, based on sample sizes between 30 and 150. Here, we investigate the variability of the coverage probability of these three and another three promising nonparametric quantile estimators with n=50(50)200,250(250)1000. The SQ estimator outperformed the HD and SV estimators for n=50 and was slightly better for n=100, whereas the SQ, HD, and SV estimators performed identically well for n≥150. The similarity of the boxplots for the SQ estimator across both distributions and sample sizes was striking.

Nonparametric Limits of Agreement for Small to Moderate Sample Sizes: A Simulation Study

The assessment of agreement in method comparison and observer variability analysis of quantitative measurements is usually done by the Bland–Altman Limits of Agreement, where the paired differences are implicitly assumed to follow a normal distribution. Whenever this assumption does not hold, the 2.5% and 97.5% percentiles are obtained by quantile estimation. In the literature, empirical quantiles have been used for this purpose. In this simulation study, we applied both sample, subsampling, and kernel quantile estimators, as well as other methods for quantile estimation to sample sizes between 30 and 150 and different distributions of the paired differences. The performance of 15 estimators in generating prediction intervals was measured by their respective coverage probability for one newly generated observation. Our results indicated that sample quantile estimators based on one or two order statistics outperformed all of the other estimators and they can be used for deriving nonparametric Limits of Agreement. For sample sizes exceeding 80 observations, more advanced quantile estimators, such as the Harrell–Davis and estimators of Sfakianakis–Verginis type, which use all of the observed differences, performed likewise well, but may be considered intuitively more appealing than simple sample quantile estimators that are based on only two observations per quantile.

Confidence Interval Estimation for Precipitation Quantiles Based on Principle of Maximum Entropy

Confidence interval of is an interval corresponding to a specified confidence and including the true value. It can be used to describe the precision of a statistical quantity and quantify its uncertainty. Although the principle of maximum entropy (POME) has been used for a variety of applications in hydrology, the confidence intervals of the POME quantile estimators have not been available. In this study, the calculation formulas of asymptotic variances and confidence intervals of quantiles based on POME for Gamma, Pearson type 3 (P3) and Extreme value type 1 (EV1) distributions were derived. Monte Carlo Simulation experiments were performed to evaluate the performance of derived formulas for finite samples. Using four data sets for annual precipitation at the Weihe River basin in China, the derived formulas were applied for calculating the variances and confidence intervals of precipitation quantiles for different return periods and the results were compared with those of the methods of moments (MOM) and of maximum likelihood (ML) method. It is shown that POME yields the smallest standard errors and the narrowest confidence intervals of quantile estimators among the three methods, and can reduce the uncertainty of quantile estimators