Mind the gap: performance metric evaluation in brain-age prediction

Estimating age based on neuroimaging-derived data has become a popular approach to developing markers for brain integrity and health. While a variety of machine-learning algorithms can provide accurate predictions of age based on brain characteristics, there is significant variation in model accuracy reported across studies. We predicted age based on neuroimaging data in two population-based datasets, and assessed the effects of age range, sample size, and age-bias correction on the model performance metrics r, R2, Root Mean Squared Error (RMSE), and Mean Absolute Error (MAE). The results showed that these metrics vary considerably depending on cohort age range; r and R2 values are lower when measured in samples with a narrower age range. RMSE and MAE are also lower in samples with a narrower age range due to smaller errors/brain age delta values when predictions are closer to the mean age of the group. Across subsets with different age ranges, performance metrics improve with increasing sample size. Performance metrics further vary depending on prediction variance as well as mean age difference between training and test sets, and age-bias corrected metrics indicate high accuracy - also for models showing poor initial performance. In conclusion, performance metrics used for evaluating age prediction models depend on cohort and study-specific data characteristics, and cannot be directly compared across different studies. Since age-bias corrected metrics in general indicate high accuracy, even for poorly performing models, inspection of uncorrected model results provides important information about underlying model attributes such as prediction variance.

Missing Observations in Split-Plot Central Composite Designs: The Loss in Relative A-, G-, and V- Efficiency

The trace (A), maximum average prediction variance (G), and integrated average prediction variance (V) criteria are experimental design evaluation criteria, which are based on precision of estimates of parameters and responses. Central Composite Designs(CCD) conducted within a split-plot structure (split-plot CCDs) consists of factorial (𝑓), whole-plot axial (𝛼), subplot axial (𝛽), and center (𝑐) points, each of which play different role in model estimation. This work studies relative A-, G- and V-efficiency losses due to missing pairs of observations in split-plot CCDs under different ratios (d) of whole-plot and sub-plot error variances. Three candidate designs of different sizes were considered and for each of the criteria, relative efficiency functions were formulated and used to investigate the efficiency of each of the designs when some observations were missing relative to the full one. Maximum A-efficiency losses of 19.1, 10.6, and 15.7% were observed at 𝑑 = 0.5, due to missing pairs 𝑓𝑓, 𝛽𝛽, and 𝑓𝛽, respectively, indicating a negative effect on the precision of estimates of model parameters of these designs. However, missing observations of the pairs- 𝑐𝑐, 𝛼𝛼, 𝛼𝑐, 𝑓𝑐, and 𝑓𝛼 did not exhibit any negative effect on these designs' relative A-efficiency. Maximum G- and Vefficiency losses of 10.1,16.1,0.1% and 0.1, 1.1, 0.2%, were observed, respectively, at 𝑑 = 0.5, when the pairs- 𝑓𝑓, 𝛽𝛽, 𝑐𝑐, were missing, indicating a significant increase in the designs' maximum and average variances of prediction. In all, the efficiency losses become insignificant as d increases. Thus, the study has identified the positive impact of correlated observations on efficiency of experimental designs. Keywords: Missing Observations, Efficiency Loss, Prediction variance

Scaled Prediction Variances of Equiradial Design under Changing Design Sizes, Axial Distances and Center Runs

The aim of every design choice is to minimize the prediction error, especially at every location of the design space, thus, it is important to measure the error at all locations in the design space ranging from the design center (origin) to the perimeter (distance from the origin). The measure of the errors varies from one design type to another and considerably the distance from the design center. Since this measure is affected by design sizes, it is ideal to scale the variance for the purpose of model comparison. Therefore, we have employed the Scaled Prediction Variance and D – optimality criterion to check the behavior of equiradial designs and compare them under varying axial distances, design sizes and center points. The following similarities were observed: (i) increasing the design radius (axial distance) of an equiradial design changes the maximum determinant of the information matrix by five percent of the new axial distance (5% of 1.414 = 0.07) see Table 3. (ii) increasing the nc center runs pushes the maximum SPV(x) to the furthest distance from the design center (0 0) (iii) changing the design radius changes the location in the design region with maximum SPV(x) by a multiple of the change and (iv) changing the design radius also does not change the maximum SPV(x) at different radial points and center runs . Based on the findings of this research, we therefore recommend consideration of equiradial designs with only two center runs in order to maximize the determinant of the information matrix and minimize the scaled prediction variances.

Optimal Prediction Variance Capabilities of Inscribed Central Composite Designs

This study looks at the effects of replication on prediction variance performances of inscribe central composite design especially those without replication on the factorial and axial portion (ICCD1), inscribe central composite design with replicated axial portion (ICCD2) and inscribe central composite design whose factorial portion is replicated (ICCD3). The G-optimal, I-optimal and FDS plots were used to examine these designs. Inscribe central composite design without replicated factorial and axial portion (ICCD1) has a better maximum scaled prediction variance (SPV) at factors k = 2 to 4 while inscribe central composite design with replicated factorial portion (ICCD3) has a better maximum and average SPV at 5 and 6 factor levels. The fraction of design space (FDS) plots show that the inscribe central composite design is superior to ICCD3 and inscribe central composite design with replicated axial portion (ICCD2) from 0.0 to 0.5 of the design space while inscribe central composite design with replicated factorial portion (ICCD3) is superior to ICCD1 and ICCD2 from 0.6 to 1.0 of the design space for factors k = 2 to 4.

Estimating Effect of Additional Sample on Uncertainty Reduction in Reliability Analysis Using Gaussian Process

An approach is proposed to quantify the uncertainty in probability of failure using a Gaussian process (GP) and to estimate uncertainty change before actually adding samples to GP. The approach estimates the coefficient of variation (CV) of failure probability due to prediction variance of GP. The CV is estimated using single-loop Monte Carlo simulation (MCS), which integrates the probabilistic classification function while replacing expensive multi-loop MCS. The methodology ensures a conservative estimate of CV, in order to compensate for sampling uncertainty in MCS. Uncertainty change is estimated by adding a virtual sample from the current GP and calculating the change in CV, which is called expected uncertainty change (EUC). The proposed method can help adaptive sampling schemes to determine when to stop before adding a sample. In numerical examples, the proposed method is used in conjunction with the efficient local reliability analysis to calculate the reliability of analytical function as well as the battery drop test simulation. It is shown that the EUC converges to the true uncertainty change as the model becomes accurate.