scholarly journals Prevalence-Value-Accuracy Plots: A New Method for Comparing Diagnostic Tests Based on Misclassification Costs

1999 ◽  
Vol 45 (7) ◽  
pp. 934-941 ◽  
Author(s):  
Alan T Remaley ◽  
Maureen L Sampson ◽  
James M DeLeo ◽  
Nancy A Remaley ◽  
Beriuse D Farsi ◽  
...  
Keyword(s):  
Diagnostic Test ◽  
Test Performance ◽  
Diagnostic Tests ◽  
Decision Threshold ◽  
Area Index ◽  
Misclassification Costs ◽  
Plot Analysis ◽  
Optimum Decision ◽  
Diagnostic Test Performance ◽  
The Cost

Abstract The clinical accuracy of diagnostic tests commonly is assessed by ROC analysis. ROC plots, however, do not directly incorporate the effect of prevalence or the value of the possible test outcomes on test performance, which are two important factors in the practical utility of a diagnostic test. We describe a new graphical method, referred to as a prevalence-value-accuracy (PVA) plot analysis, which includes, in addition to accuracy, the effect of prevalence and the cost of misclassifications (false positives and false negatives) in the comparison of diagnostic test performance. PVA plots are contour plots that display the minimum cost attributable to misclassifications (z-axis) at various optimum decision thresholds over a range of possible values for prevalence (x-axis) and the unit cost ratio (UCR; y-axis), which is an index of the cost of a false-positive vs a false-negative test result. Another index based on the cost of misclassifications can be derived from PVA plots for the quantitative comparison of test performance. Depending on the region of the PVA plot that is used to calculate the misclassification cost index, it can potentially lead to a different interpretation than the ROC area index on the relative value of different tests. A PVA-threshold plot, which is a variation of a PVA plot, is also described for readily identifying the optimum decision threshold at any given prevalence and UCR. In summary, the advantages of PVA plot analysis are the following: (a) it directly incorporates the effect of prevalence and misclassification costs in the analysis of test performance; (b) it yields a quantitative index based on the costs of misclassifications for comparing diagnostic tests; (c) it provides a way to restrict the comparison of diagnostic test performance to a clinically relevant range of prevalence and UCR; and (d) it can be used to directly identify an optimum decision threshold based on prevalence and misclassification costs.

The Use of Information Graphs to Evaluate and Compare Diagnostic Tests

2002 ◽  
Vol 41 (02) ◽  
pp. 114-118 ◽  
Author(s):  
W. A. Benish
Keyword(s):  
Diagnostic Test ◽  
Relative Entropy ◽  
Test Performance ◽  
Diagnostic Tests ◽  
Critical Role ◽  
Diagnostic Information ◽  
Pretest Probability ◽  
Operating Characteristics ◽  
Diagnostic Uncertainty ◽  
Diagnostic Test Performance

Summary Objectives: The purpose of this communication is to demonstrate the use of “information graphs” as a means of characterizing diagnostic test performance. Methods: Basic concepts in information theory allow us to quantify diagnostic uncertainty and diagnostic information. Given the probabilities of the diagnoses that can explain a patient’s condition, the entropy of that distribution is a measure of our uncertainty about the diagnosis. The relative entropy of the posttest probabilities with respect to the pretest probabilities quantifies the amount of information gained by diagnostic testing. Mutual information is the expected value of relative entropy and, hence, provides a measure of expected diagnostic information. These concepts are used to derive formulas for calculating diagnostic information as a function of pretest probability for a given pair of test operating characteristics. Results: Plots of diagnostic information as a function of pretest probability are constructed to evaluate and compare the performance of three tests commonly used in the diagnosis of coronary artery disease. The graphs illustrate the critical role that the pretest probability plays in determining diagnostic test information. Conclusions: Information graphs summarize diagnostic test performance and offer a way to evaluate and compare diagnostic tests.

Estimation of Performance and Sequential Selection of Diagnostic Tests in Patients With Lung Lesions Suspicious for Cancer

2002 ◽  
Vol 126 (1) ◽  
pp. 19-27
Author(s):  
Dana Marie Grzybicki ◽  
Thomas Gross ◽  
Kim R. Geisinger ◽  
Stephen S. Raab
Keyword(s):  
Diagnostic Test ◽  
Test Performance ◽  
Diagnostic Tests ◽  
Needle Aspiration ◽  
Selection Strategy ◽  
Test Selection ◽  
Lung Lesions ◽  
Sequential Selection ◽  
Test Ordering ◽  
Selection Of

Abstract Context.—Measuring variation in clinician test ordering behavior for patients with similar indications is an important focus for quality management and cost containment. Objective.—To obtain information from physicians and nonphysicians regarding their test-ordering behavior and their knowledge of test performance characteristics for diagnostic tests used to work up patients with lung lesions suspicious for cancer. Design.—A self-administered, voluntary, anonymous questionnaire was distributed to 452 multiple-specialty physicians and 500 nonphysicians in academic and private practice in Pennsylvania, Iowa, and North Carolina. Respondents indicated their estimates of test sensitivities for multiple tests used in the diagnosis of lung lesions and provided their test selection strategy for case simulations of patients with solitary lung lesions. Data were analyzed using descriptive statistics and the χ2 test. Results.—The response rate was 11.2%. Both physicians and nonphysicians tended to underestimate the sensitivities of all minimally invasive tests, with the greatest underestimations reported for sputum cytology and transthoracic fine-needle aspiration biopsy. There was marked variation in sequential test selection for all the case simulations and no association between respondent perception of test sensitivity and their selection of first diagnostic test. Overall, the most frequently chosen first diagnostic test was bronchoscopy. Conclusions.—Physicians and nonphysicians tend to underestimate the performance of diagnostic tests used to evaluate solitary lung lesions. However, their misperceptions do not appear to explain the wide variation in test-ordering behavior for patients with lung lesions suspicious for cancer.

Cost-Sensitive Learning in Medicine

2011 ◽  
pp. 57-75
Author(s):  
Alberto Freitas ◽  
Pavel Brazdil ◽  
Altamiro Costa-Pereira
Keyword(s):  
Diagnostic Tests ◽  
False Negative ◽  
Cost Sensitive Learning ◽  
Safety Issues ◽  
Misclassification Costs ◽  
Misclassification Errors ◽  
Cost Sensitive Classification ◽  
Human Costs ◽  
The Cost ◽  
Specific Diagnostic Test

This chapter introduces cost-sensitive learning and its importance in medicine. Health managers and clinicians often need models that try to minimize several types of costs associated with healthcare, including attribute costs (e.g. the cost of a specific diagnostic test) and misclassification costs (e.g. the cost of a false negative test). In fact, as in other professional areas, both diagnostic tests and its associated misclassification errors can have significant financial or human costs, including the use of unnecessary resource and patient safety issues. This chapter presents some concepts related to cost-sensitive learning and cost-sensitive classification and its application to medicine. Different types of costs are also present, with an emphasis on diagnostic tests and misclassification costs. In addition, an overview of research in the area of cost-sensitive learning is given, including current methodological approaches. Finally, current methods for the cost-sensitive evaluation of classifiers are discussed.

Mutual Information as an Index of Diagnostic Test Performance

2003 ◽  
Vol 42 (03) ◽  
pp. 260-264 ◽  
Author(s):  
W. A. Benish
Keyword(s):  
Mutual Information ◽  
Diagnostic Test ◽  
Test Performance ◽  
Random Variable ◽  
Disease State ◽  
Necessary Condition ◽  
Conditional Probabilities ◽  
Amount Of Information ◽  
Diagnostic Test Performance ◽  
Test Result

Summary Objectives: This paper demonstrates that diagnostic test performance can be quantified as the average amount of information the test result (R) provides about the disease state (D). Methods: A fundamental concept of information theory, mutual information, is directly applicable to this problem. This statistic quantifies the amount of information that one random variable contains about another random variable. Prior to performing a diagnostic test, R and D are random variables. Hence, their mutual information, I(D;R), is the amount of information that R provides about D. Results: I(D;R) is a function of both 1) the pretest probabilities of the disease state and 2) the set of conditional probabilities relating each possible test result to each possible disease state. The area under the receiver operating characteristic curve (AUC) is a popular measure of diagnostic test performance which, in contrast to I(D;R), is independent of the pretest probabilities; it is a function of only the set of conditional probabilities. The AUC is not a measure of diagnostic information. Conclusions: Because I(D;R) is dependent upon pretest probabilities, knowledge of the setting in which a diagnostic test is employed is a necessary condition for quantifying the amount of information it provides. Advantages of I(D;R) over the AUC are that it can be calculated without invoking an arbitrary curve fitting routine, it is applicable to situations in which multiple diagnoses are under consideration, and it quantifies test performance in meaningful units (bits of information).

scholarly journals Summarizing Studies of Diagnostic Test Performance

2003 ◽  
Vol 49 (11) ◽  
pp. 1783-1784 ◽  
Author(s):  
Victor M Montori ◽  
Gordon H Guyatt
Keyword(s):  
Diagnostic Test ◽  
Test Performance ◽  
Diagnostic Test Performance
Sign in / Sign up

Export Citation Format

Share Document