An approach to standardizing human performance assessment.

Selection of a measure of effectiveness (MOE) (a mathematical function) and using that measure to evaluate performance demonstrations (or exercises, or experimental trials) without first testing the measure, typically results in a disagreement between two ways of assigning effectiveness scores to each performance demonstration. The two ways of assigning effectiveness scores to each performance demonstration are: effectiveness scores assigned directly by the investigator and effectiveness scores assigned by the MOE selected by the investigator. The disagreement often exists even when comparing the rank ordering of the two sets of scored performance demonstrations. A disagreement between the two methods means that one method, possibly both, are not correct. The direct assignment of effectiveness scores to each performance demonstration constitutes a test of the MOE. In this paper, we argue that test is typically not conducted and if it were, the MOE (existing untested MOE's) would likely fail the test. We also argue that the investigator should not select an MOE but rather should have an authority (SME) score performance demonstrations and then synthesize an MOE that will pass the test. A method for synthesizing the MOE is presented.

Download Full-text

The Criterion Task Set: Validation and Application

Proceedings of the Human Factors Society Annual Meeting ◽

10.1177/154193128603000114 ◽

1986 ◽

Vol 30 (1) ◽

pp. 57-57

Author(s):

Clark A Shingledecker

Keyword(s):

Performance Assessment ◽

Large Scale ◽

Human Performance ◽

Scale Validation ◽

Relative Sensitivity ◽

Assessment System ◽

Assessment Instrument ◽

Performance Tasks ◽

Task Set ◽

Criterion Task

The Criterion Task Set (CTS) is a battery of performance tasks which was developed at the Air Force Armstrong Aerospace Medical Research Laboratory. Based on an information processing stage/resource model of human performance, the CTS was designed to evaluate the relative sensitivity, diagnosticity and intrusiveness of available measures of operator workload. It has also been employed as a performance assessment instrument to evaluate the effects of stressors on hypothesized independent sources of performance capability. Since the completion of original developmental research and the implementation of the CTS in a standard hardware/software system, a number of researchers have employed the battery in applied human performance studies and in efforts which have contributed to its further refinement. The objectives of this symposium are to present accounts of six of these research projects and to provide a forum for individuals who are currently using the CTS or who are interested in potential applications of this performance assessment system. The papers presented in the symposium include a report of a large-scale validation project which has formed the basis for a CTS data base (Schlegel, Gilliland and Schlegel), as well as a study aimed at improvement of one of the tasks comprising the battery (Eggemeier and Amell). The remaining four papers describe applications of the CTS to the investigation of physiological (Wilson and McCloskey) and subjective (Acton, Reid and Perez) workload metrics, and to the study of individual differences (Gilliland, Schlegel and Dannels) and subjective arousal states (Kimball and Pond).

Download Full-text