Test-Retest Reliability is not a Measure of Reliability or Stability: A Friendly Reminder

We argue that the test-retest reliability coefficient, which is the correlation between a measurement and a repeated measurement using the same diagnostic instrument in the same sample (sometimes referred to as repeatability or falsely referred to as stability), is by itself not an appropriate measure of the reliability of the diagnostic instrument or of the stability of the construct in question. In combination with an actual coefficient of reliability such as Cronbach’s alpha, the test-retest reliability coefficient can be used to estimate and compare the stabilities of constructs using a procedure based on the correction for attenuation. However, results from a simulation study showed that classically constructed confidence intervals for the estimator exhibit under-coverage and thus cannot be interpreted correctly.

Modifying Spearman’s Attenuation Equation to Yield Partial Corrections for Measurement Error—With Application to Sample Size Calculations

Spearman’s correction for attenuation (measurement error) corrects a correlation coefficient for measurement errors in either-or-both of two variables, and follows from the assumptions of classical test theory. Spearman’s equation removes all measurement error from a correlation coefficient which translates into “increasing the reliability of either-or-both of two variables to 1.0.” In this inquiry, Spearman’s correction is modified to allow partial removal of measurement error from either-or-both of two variables being correlated. The practical utility of this partial correction is demonstrated in its use to explore increasing the power of statistical tests by increasing sample size versus increasing the reliability of the dependent variable for an experiment. Other applied uses are mentioned.

Bayesian Correction for Attenuation of Correlation in Multi-Trial Spike Count Data

When correlation is measured in the presence of noise, its value is decreased. In single-neuron recording experiments, for example, the correlation of selectivity indices in a pair of tasks may be assessed across neurons, but, because the number of trials is limited, the measured index values for each neuron will be noisy. This attenuates the correlation. A correction for such attenuation was proposed by Spearman more than 100 yr ago, and more recent work has shown how confidence intervals may be constructed to supplement the correction. In this paper, we propose an alternative Bayesian correction. A simulation study shows that this approach can be far superior to Spearman's, both in accuracy of the correction and in coverage of the resulting confidence intervals. We demonstrate the usefulness of this technology by applying it to a set of data obtained from the frontal cortex of a macaque monkey while performing serial order and variable reward saccade tasks. There the correction results in a substantial increase in the correlation across neurons in the two tasks.

Explaining the Weak Relationship Between Job Performance and Ratings of Job Performance

Ratings of job performance are widely viewed as poor measures of job performance. Three models of the performance–performance rating relationship offer very different explanations and solutions for this seemingly weak relationship. One-factor models suggest that measurement error is the main difference between performance and performance ratings and they offer a simple solution—that is, the correction for attenuation. Multifactor models suggest that the effects of job performance on performance ratings are often masked by a range of systematic nonperformance factors that also influence these ratings. These models suggest isolating and dampening the effects of these nonperformance factors. Mediated models suggest that intentional distortions are a key reason that ratings often fail to reflect ratee performance. These models suggest that raters must be given both the tools and the incentive to perform well as measurement instruments and that systematic efforts to remove the negative consequences of giving honest performance ratings are needed if we hope to use performance ratings as serious measures of job performance.

A Variational Scheme for Retrieving Rainfall Rate and Hail Reflectivity Fraction from Polarization Radar

Polarization radar offers the promise of much more accurate rainfall-rate R estimates than are possible from radar reflectivity factor Z alone, not only by better characterization of the drop size distribution, but also by more reliable correction for attenuation and the identification of hail. However, practical attempts to implement retrieval algorithms have been hampered by the difficulty in coping with the inherent noise in the polarization parameters. In this paper, a variational retrieval scheme is described that overcomes these problems by employing a forward model for differential reflectivity Zdr and differential phase shift ϕdp and iteratively refining the coefficient a in the relationship Z = aRb such that the difference between the forward model and the measurements is minimized in a least squares sense. Two methods are used to ensure that a varies smoothly in both range and azimuth. In range, a is represented by a set of cubic-spline basis functions; in azimuth, the retrieval at one ray is used as a constraint on the next. The result of this smoothing is that the retrieval is tolerant of random errors in Zdr of up to 1 dB and in ϕdp of up to 5°. Correction for attenuation is achieved simply and effectively by including its effects in the forward model. If hail is present then the forward model is unable to match the observations of Zdr and ϕdp simultaneously. This enables a first pass of the retrieval to be used to identify the radar pixels that contain hail, followed by a second pass in which the fraction of the Z in those gates that is due to hail is retrieved, this time with the scheme being able to forward-model both Zdr and ϕdp accurately. The scheme is tested on S-band radar data from southern England in cases of rain, spherical hail, oblate hail, and mixtures of rain and hail. It is found to be robust and stable, even in the presence of differential phase shift on backscatter.