Recent proposals to estimate transient error within the classical test theory tradition

Reliability is conceptually defined in terms of consistency across test occasions but coefficient alpha, the most popular reliability estimation method, precludes the examination of such consistency. Three recent proposals to estimate transient error separately within a classical test theory tradition, and the results that they have yielded are reviewed. The merits of these proposals are compared with those of generalisability theory which differentiates between different sources of error variation. Although the procedures reviewed cannot match the advantages of generalisability theory, they may be sufficient in many applications.

Download Full-text

Reliability

10.1093/med:psych/9780198527565.003.0015 ◽

2015 ◽

Author(s):

Susanne Hempel

Keyword(s):

Correlation Coefficient ◽

Internal Consistency ◽

Classical Test Theory ◽

Test Theory ◽

Coefficient Alpha ◽

Parallel Test ◽

Reliability Measure ◽

Classical Test

This chapter discusses reliability. It outlines the nature and purpose of reliability, classical test theory, measures of reliability (measure orientated reliability, parallel test, and test-retest) as well as internal consistency, inter-item correlation, coefficient alpha, and categorical judgements.

Download Full-text

ON THE INTEGRITY OF RELIABILITY ESTIMATION IN CLASSICAL TEST THEORY: THE CASE FOR AN ADDITIVE COEFFICIENT OF STABILITY

Psychological Reports ◽

10.2466/pr0.103.6.545-565 ◽

2008 ◽

Vol 103 (6) ◽

pp. 545

Author(s):

GILBERT BECKER

Keyword(s):

Classical Test Theory ◽

Reliability Estimation ◽

Test Theory ◽

Classical Test

Download Full-text

Controlling Decremental and Inflationary Effects in Reliability Estimation Resulting from Violations of Assumptions

Psychological Reports ◽

10.2466/pr0.2001.89.2.403 ◽

2001 ◽

Vol 89 (2) ◽

pp. 403-424 ◽

Cited By ~ 4

Author(s):

Gilbert Becker

Keyword(s):

Opposite Direction ◽

Measurement Errors ◽

Classical Test Theory ◽

Reliability Estimation ◽

Test Theory ◽

Measuring Instruments ◽

Measuring Instrument ◽

Correlated Errors ◽

Classical Test ◽

Practical Level

Two assumptions in classical test theory, essential tau-equivalence and independence of measurement errors, when violated may produce attenuated or inflated estimates of reliability, respectively. Inflation stemming from correlated errors can be controlled by a procedure in which systematically created equivalent halves of a given measuring instrument are administered across two occasions. When poor approximations to equivalent halves are constructed for this purpose, however, distortion in the opposite direction may result, being sometimes quite large when measuring instruments are not essentially tau-equivalent (or, at the practical level, unidimensional). The nature of these decrements are discussed and illustrated, and a number of procedures for eliminating them introduced.

Download Full-text

On the Integrity of Reliability Estimation in Classical Test Theory: The Case for an Additive Coefficient of Stability

Psychological Reports ◽

10.2466/pr0.103.2.545-565 ◽

2008 ◽

Vol 103 (2) ◽

pp. 545-565

Author(s):

Gilbert Becker

Keyword(s):

Classical Test Theory ◽

Theory Development ◽

Error Component ◽

Reliability Estimation ◽

Test Theory ◽

Extended Model ◽

Classical Test ◽

Scale Reliability ◽

Recent Developments ◽

Alternate Forms

This article addresses deficiencies in the most widely used estimators of reliability and draws attention to the reason that this issue is important. Accurate calibration of relationships between constructs is critical to theory development. Unless workers have accurate estimates of scale reliability, accurate estimates of those relationships will not be forthcoming because the classical disattenuation formula requires them. This article shows that classical test theory can easily accommodate the delineation of its error component E in test scores into two sources, inconsistency across content ( E1) and inconsistency across time ( E2). Viewed from this extended model, the alternate forms approach to reliability estimation is complete in that it gauges simultaneously both sources of error. Because that approach is rarely used today for that purpose, the integrity of estimation has been lost. In its place arose estimators of partial reliability—those for estimating generalizability over one medium or the other, but not both, thereby precluding the additivity of error components. Recent developments promise to restore the integrity of the alternate forms approach without the need for alternate forms and suggest an additive alternative to the current nonadditive coefficient of stability.

Download Full-text