scholarly journals Cognitive screening instruments for dementia: comparing metrics of test limitation

2020 ◽  
Author(s):  
Andrew J Larner
Keyword(s):  
Test Performance ◽  
High Sensitivity ◽  
Misclassification Rate ◽  
Test Accuracy ◽  
Screening Instruments ◽  
Cognitive Screening ◽  
Net Benefit ◽  
Likelihood Ratios ◽  
Benefit Ratio ◽  
Misclassification Rates

AbstractCognitive screening instruments (CSIs) for dementia and mild cognitive impairment are usually characterised in terms of measures of discrimination such as sensitivity, specificity, and likelihood ratios. However, CSIs also have limitations. Several metrics exist which may be used to denote test limitations but they are seldom examined. Data from several pragmatic test accuracy studies of CSIs were interrogated to calculate various measures of limitation, namely: misclassification rate; net harm to net benefit ratio; and the likelihood to be diagnosed or misdiagnosed. Intra- and inter-test performance for measures of discrimination and limitation were compared. The study found that some tests with very high sensitivity but low specificity for dementia fared poorly on measures of limitation, with high misclassification rates, low net harm to net benefit ratios, and low likelihoods to be diagnosed or misdiagnosed; some had likelihoods favouring misdiagnosis over diagnosis. Tests with a better balance of sensitivity and specificity fared better on measures of limitation. When choosing which CSIs to administer, measures of test limitation should be considered as well as measures of test discrimination. Although high test sensitivity may be desirable to avoid false negatives, false positives also have a cost. Identification of tests having high misclassification rate, low net harm to net benefit ratio, and low likelihood to be diagnosed or misdiagnosed, may have implications for their use in clinical practice.

scholarly journals Cognitive screening instruments for dementia: comparing metrics of test limitation

2021 ◽  
Vol 15 (4) ◽  
pp. 458-463
Author(s):  
Andrew J. Larner
Keyword(s):  
Cognitive Impairment ◽  
High Sensitivity ◽  
Misclassification Rate ◽  
Test Accuracy ◽  
Screening Instruments ◽  
Cognitive Screening ◽  
Net Benefit ◽  
Likelihood Ratios ◽  
Sensitivity Specificity ◽  
State Examination

ABSTRACT Cognitive screening instruments (CSIs) for dementia and mild cognitive impairment are usually characterized in terms of measures of discrimination such as sensitivity, specificity, and likelihood ratios, but these CSIs also have limitations. Objective: The aim of this study was to calculate various measures of test limitation for commonly used CSIs, namely, misclassification rate (MR), net harm/net benefit ratio (H/B), and the likelihood to be diagnosed or misdiagnosed (LDM). Methods: Data from several previously reported pragmatic test accuracy studies of CSIs (Mini-Mental State Examination, the Montreal Cognitive Assessment, Mini-Addenbrooke’s Cognitive Examination, Six-item Cognitive Impairment Test, informant Ascertain Dementia 8, Test Your Memory test, and Free-Cog) undertaken in a single clinic were reanalyzed to calculate and compare MR, H/B, and the LDM for each test. Results: Some CSIs with very high sensitivity but low specificity for dementia fared poorly on measures of limitation, with high MRs, low H/B, and low LDM; some had likelihoods favoring misdiagnosis over diagnosis. Tests with a better balance of sensitivity and specificity fared better on measures of limitation. Conclusions: When deciding which CSI to administer, measures of test limitation as well as measures of test discrimination should be considered. Identification of CSIs with high MR, low H/B, and low LDM, may have implications for their use in clinical practice.

MACE versus MoCA: equivalence or superiority? Pragmatic diagnostic test accuracy study

2017 ◽  
Vol 29 (6) ◽  
pp. 931-937 ◽  
Author(s):  
A.J. Larner
Keyword(s):  
Secondary Care ◽  
Characteristic Curve ◽  
High Sensitivity ◽  
Screening Instrument ◽  
Test Accuracy ◽  
Cognitive Screening ◽  
Net Benefit ◽  
Dementia Diagnosis ◽  
Predictive Values ◽  
Accuracy Study

ABSTRACTBackground:The Mini-Addenbrooke's Cognitive Examination (MACE) is a new brief cognitive screening instrument for dementia and mild cognitive impairment (MCI). Historical data suggest that MACE may be comparable to the Montreal Cognitive Assessment (MoCA), a well-established cognitive screening instrument, in secondary care settings, but no head-to-head study has been reported hitherto.Methods:A pragmatic diagnostic accuracy study of MACE and MoCA was undertaken in consecutive patients referred over the course of one year to a neurology-led Cognitive Function Clinic, comparing their performance for the diagnosis of dementia and MCI using various test metrics.Results:In a cohort of 260 patients with dementia and MCI prevalence of 17% and 29%, respectively, both MACE and MoCA were quick and easy to use and acceptable to patients. Both tests had high sensitivity (>0.9) and large effect sizes (Cohen's d) for diagnosis of both dementia and MCI but low specificity and positive predictive values. Area under the receiver operating characteristic curve was excellent for dementia diagnosis (both >0.9) but less good for MCI (MoCA good and MACE fair). In contrast, weighted comparison suggested test equivalence for dementia diagnosis but with a slight net benefit for MACE for MCI diagnosis.Conclusions:MACE is an acceptable and accurate test for the assessment of cognitive problems, with performance comparable to MoCA. MACE appears to be a viable alternative to MoCA for testing patients with cognitive complaints in a secondary care setting.

MACE for the Diagnosis of Dementia and MCI: 3-Year Pragmatic Diagnostic Test Accuracy Study

2018 ◽  
Vol 45 (5-6) ◽  
pp. 300-307 ◽  
Author(s):  
John C. Williamson ◽  
Andrew J. Larner
Keyword(s):  
Diagnostic Test ◽  
Cognitive Disorders ◽  
High Sensitivity ◽  
Diagnostic Test Accuracy ◽  
Test Accuracy ◽  
Cognitive Screening ◽  
Predictive Values ◽  
Diagnosis Of Dementia ◽  
Accuracy Study ◽  
Secondary Care Setting

Background/Aims: The Mini-Addenbrooke’s Cognitive Examination (MACE) is a relatively new short cognitive screening instrument for the detection of patients with dementia and mild cognitive impairment (MCI). Few studies of the MACE have been reported hitherto. The aim of this study was to undertake a pragmatic diagnostic test accuracy study of MACE in a large cohort of patients seen in a dedicated cognitive disorders clinic. Methods: MACE was administered to consecutive patients referred to a neurology-led Cognitive Function Clinic over the course of 3 years to assess its performance for the diagnosis of dementia and MCI using various test metrics. Results: In a cohort of 599 patients, the prevalence of dementia and MCI by criterion diagnosis was 0.17 and 0.29, respectively. MACE had a high sensitivity (> 0.9) and negative predictive values (> 0.8) with large effect sizes (Cohen’s d > 1) for the diagnosis of both dementia and MCI but a low specificity (< 0.5) and positive predictive values (≤0.5). Conclusion: MACE is an acceptable test for the assessment of cognitive complaints in a secondary care setting with good metrics for identifying cases of both dementia and MCI.

Optimising the Cutoffs of Cognitive Screening Instruments in Pragmatic Diagnostic Accuracy Studies: Maximising Accuracy or the Youden Index?

2015 ◽  
Vol 39 (3-4) ◽  
pp. 167-175 ◽  
Author(s):  
Andrew J. Larner
Keyword(s):  
Diagnostic Accuracy ◽  
Alternative Methods ◽  
Youden Index ◽  
Test Accuracy ◽  
Screening Instruments ◽  
Cognitive Screening ◽  
Optimal Method ◽  
Index Method ◽  
State Examination ◽  
Addenbrooke’S Cognitive Examination

Background/Aims: The optimal method of establishing test cutoffs or cutpoints for cognitive screening instruments (CSIs) is uncertain. Of the available methods, two base cutoffs on either the maximal test accuracy or the maximal Youden index. The aim of this study was to compare the effects of using these alternative methods of establishing cutoffs. Methods: Datasets from three pragmatic diagnostic accuracy studies which examined the Mini-Mental State Examination (MMSE), the Addenbrooke's Cognitive Examination-Revised (ACE-R), the Montreal Cognitive Assessment (MoCA), and the Test Your Memory (TYM) test were analysed to calculate test sensitivity and specificity using cutoffs based on either maximal test accuracy or the maximal Youden index. Results: For ACE-R, MoCA, and TYM, optimal cutoffs for dementia diagnosis differed from those in index studies when defined using either the maximal accuracy or the maximal Youden index method. Optimal cutoffs were higher for MMSE, MoCA, and TYM when using the maximal Youden index method and consequently more sensitive. Conclusion: Revision of the cutoffs for CSIs established in index studies may be required to optimise performance in pragmatic diagnostic test accuracy studies which more closely resemble clinical practice.

What is test accuracy? Comparing unitary accuracy metrics for cognitive screening instruments

2019 ◽  
Vol 9 (5) ◽  
pp. 277-281 ◽  
Author(s):  
Andrew J Larner
Keyword(s):  
Cognitive Impairment ◽  
Classification Accuracy ◽  
Operating Characteristic ◽  
Characteristic Curve ◽  
Test Accuracy ◽  
Correct Classification ◽  
Screening Instruments ◽  
Cognitive Screening ◽  
Operating Characteristic Curve ◽  
Accuracy Measures

Aim: To examine four different accuracy metrics for assessment of commonly used cognitive screening instruments: correct classification accuracy, area under the receiver operating characteristic curve, F measure (F) or F1 score and Matthews correlation coefficient (MCC). Methods: Raw data were extracted from test accuracy studies of Mini-Mental State Examination. Montreal Cognitive Assessment, Mini-Addenbrooke's Cognitive Examination, Six-item Cognitive Impairment Test, informant AD8 and Free-Cog, and used to calculate the accuracy measures. Results: Each metric resulted in similar ordering of the screening instruments for diagnosis of both dementia and mild cognitive impairment. Area under the receiver operating characteristic curve gave the highest (most optimistic) and MCC the lowest (most pessimistic) accuracy value for each test examined, with correct classification accuracy and F falling between. Conclusion: All the accuracy measures examined have potential shortcomings. None can be recommended as the definitive unitary outcome measure for test accuracy studies. However, MCC has theoretical advantages and might be more widely adopted.

scholarly journals ROC plot and AUC with binary classifiers: pragmatic analysis of cognitive screening instruments

2021 ◽  
Author(s):  
Gashirai K Mbizvo ◽  
Andrew J Larner
Keyword(s):  
Test Performance ◽  
False Positive Rate ◽  
Area Under The Curve ◽  
True Positive Rate ◽  
Cognitive Status ◽  
Test Accuracy ◽  
Cognitive Screening ◽  
Continuous Scale ◽  
Positive Rate ◽  
Binary Classifiers

Receiver operating characteristic (ROC) plots are a performance graphing method showing the relative trade-off between test benefits (true positive rate) and costs (false positive rate) with the area under the curve (AUC) giving a scalar value of test performance. It has been suggested that ROC and AUC may be potentially misleading when examining binary predictors rather than continuous scales. The purpose of this study was to examine ROC plots and AUC values for two binary classifiers of cognitive status (applause sign, attended with sign), a cognitive screening instrument producing categorical data (Codex), and a continuous scale screening test (Mini-Addenbrooke's Cognitive Examination), the latter two also analysed with single fixed threshold tests. For each of these plots, AUC was calculated using different methods. The findings indicate that if categorical or continuous measures are dichotomised then the calculated AUC may be an underestimate, thus affecting screening or diagnostic test accuracy which in the context of clinical practice may prove to be misleading.

scholarly journals Mini-Cog versus Codex (cognitive disorders examination) Is there a difference?

2020 ◽  
Vol 14 (2) ◽  
pp. 128-133 ◽  
Author(s):  
Andrew J. Larner
Keyword(s):  
Cognitive Disorders ◽  
High Sensitivity ◽  
Word Recall ◽  
Screening Tests ◽  
Cognitive Screening ◽  
Net Benefit ◽  
Dementia Diagnosis ◽  
Subjective Memory ◽  
Diagnosis Of Dementia ◽  
Screening Accuracy

Abstract. Mini-Cog and Codex (cognitive disorders examination) are brief cognitive screening tests incorporating word-recall and clock drawing tests. Objective: To assess and compare the screening accuracy of Mini-Cog and Codex for diagnosis of dementia and mild cognitive impairment (MCI) in patients attending a dedicated cognitive disorders clinic. Methods: Tests were administered to a consecutive cohort of 162 patients, whose reference standard diagnoses based on clinical diagnostic criteria were dementia (44), MCI (26), and subjective memory complaint (92). Results: Both Mini-Cog and Codex had high sensitivity (>0.8) for dementia diagnosis, but Codex was more specific. For diagnosis of MCI, Mini-Cog had better sensitivity than Codex. Weighted comparisons of Mini-Cog and Codex showed only marginal net benefit for Mini-Cog for dementia diagnosis but larger net benefit for MCI diagnosis. Conclusion: In this pragmatic study both Mini-Cog and Codex were accurate brief screening tests for dementia but Mini-Cog was better for identification of MCI.

scholarly journals Codex (Cognitive Disorders Examination) Decision Tree Modified for the Detection of Dementia and MCI

Diagnostics ◽  
2019 ◽  
Vol 9 (2) ◽  
pp. 58 ◽  
Author(s):  
Besa Ziso ◽  
Andrew J. Larner
Keyword(s):  
Cognitive Impairment ◽  
Mild Cognitive Impairment ◽  
Decision Tree ◽  
Cognitive Disorders ◽  
High Sensitivity ◽  
Diagnostic Value ◽  
Word Recall ◽  
Screening Instruments ◽  
Cognitive Screening ◽  
Dementia Diagnosis

Many cognitive screening instruments are available to assess patients with cognitive symptoms in whom a diagnosis of dementia or mild cognitive impairment is being considered. Most are quantitative scales with specified cut-off values. In contrast, the cognitive disorders examination or Codex is a two-step decision tree which incorporates components from the Mini-Mental State Examination (MMSE) (three word recall, spatial orientation) along with a simplified clock drawing test to produce categorical outcomes defining the probability of dementia diagnosis and, by implication, directing clinician response (reassurance, monitoring, further investigation, immediate treatment). Codex has been shown to have high sensitivity and specificity for dementia diagnosis but is less sensitive for the diagnosis of mild cognitive impairment (MCI). We examined minor modifications to the Codex decision tree to try to improve its sensitivity for the diagnosis of MCI, based on data extracted from studies of two other cognitive screening instruments, the Montreal Cognitive Assessment and Free-Cog, which are more stringent than MMSE in their tests of delayed recall. Neither modification proved of diagnostic value for mild cognitive impairment. Possible explanations for this failure are considered.

Free-Cog: Pragmatic Test Accuracy Study and Comparison with Mini-Addenbrooke’s Cognitive Examination

2019 ◽  
Vol 47 (4-6) ◽  
pp. 254-263 ◽  
Author(s):  
Andrew J. Larner
Keyword(s):  
Cognitive Abilities ◽  
Cognitive Disorders ◽  
High Sensitivity ◽  
Test Accuracy ◽  
Net Benefit ◽  
Dementia Screening ◽  
Diagnosis Of Dementia ◽  
Accuracy Study ◽  
And Function ◽  
Addenbrooke’S Cognitive Examination

Background/Aims: Canonical definitions of the dementia construct encompass deficits in both cognition and function, but most screening instruments for possible dementia address only cognitive abilities. Free-Cog is a recently described brief screening instrument for dementia designed to address not only cognitive but also functional abilities. Methods: A pragmatic test accuracy study of Free-Cog was undertaken in consecutive patients seen over 1 year in a secondary care setting. The performance of Free-Cog for diagnosis of dementia and mild cognitive impairment (MCI) was compared to that of Mini-Addenbrooke’s Cognitive Examination (MACE). Results: In a cohort of 141 patients (prevalence of dementia and MCI 11 and 32%, respectively) both Free-Cog and MACE were quick and easy to use and acceptable to patients. Both tests had high sensitivity (1.00) and large effect sizes (Cohen’s d) for diagnosis of dementia, but Free-Cog was more specific. For diagnosis of MCI, Free-Cog lacked sensitivity (0.58) but was specific (0.81), whereas MACE was sensitive (0.91) but not specific (0.35). Weighted comparison suggested equivalence for dementia diagnosis but a net benefit for MACE regarding MCI diagnosis. Conclusion: Free-Cog is an acceptable and accurate test for dementia screening in a dedicated cognitive disorders clinic, but it appears less sensitive than MACE for the identification of MCI.

scholarly journals Communicating Risk: Developing an “Efficiency Index” for Dementia Screening Tests

2021 ◽  
Vol 11 (11) ◽  
pp. 1473
Author(s):  
Andrew J. Larner
Keyword(s):  
Correct Diagnosis ◽  
Efficiency Index ◽  
Screening Tests ◽  
Test Accuracy ◽  
Cognitive Screening ◽  
Likelihood Ratios ◽  
Dementia Prevalence ◽  
Dementia Screening ◽  
Potential Benefits ◽  
State Examination

Diagnostic and screening tests may have risks such as misdiagnosis, as well as the potential benefits of correct diagnosis. Effective communication of this risk to both clinicians and patients can be problematic. The purpose of this study was to develop a metric called the “efficiency index” (EI), defined as the ratio of test accuracy and inaccuracy, to evaluate screening tests for dementia. This measure was compared with a previously described “likelihood to be diagnosed or misdiagnosed” (LDM), also based on “numbers needed” metrics. Datasets from prospective pragmatic test accuracy studies examining four brief cognitive screening instruments (Mini-Mental State Examination; Montreal Cognitive Assessment; Mini-Addenbrooke’s Cognitive Examination (MACE); and Free-Cog) were analysed to calculate values for EI and LDM, and to examine their variation with test cut-off for MACE and dementia prevalence. EI values were also calculated using a modification of McGee’s heuristic for the simplification of likelihood ratios to estimate percentage change in diagnostic probability. The findings indicate that EI is easier to calculate than LDM and, unlike LDM, may be classified either qualitatively or quantitatively in a manner similar to likelihood ratios. EI shows the utility or inutility of diagnostic and screening tests, illustrating the inevitable trade-off between diagnosis and misdiagnosis. It may be a useful metric to communicate risk in a way that is easily intelligible for both clinicians and patients.

Sign in / Sign up

Export Citation Format

Share Document