scholarly journals Measuring metacognitive performance: type 1 performance dependence and test-retest reliability

2021 ◽  
Vol 2021 (1) ◽  
Author(s):  
Matthias Guggenmos
Keyword(s):  
Performance Measures ◽  
Performance Measure ◽  
Human Cognition ◽  
Total N ◽  
Retest Reliability ◽  
Simultaneous Choice ◽  
Highly Sensitive ◽  
Primary Type ◽  
Test Retest Reliability

Abstract Research on metacognition—thinking about thinking—has grown rapidly and fostered our understanding of human cognition in healthy individuals and clinical populations. Of central importance is the concept of metacognitive performance, which characterizes the capacity of an individual to estimate and report the accuracy of primary (type 1) cognitive processes or actions ensuing from these processes. Arguably one of the biggest challenges for measures of metacognitive performance is their dependency on objective type 1 performance, although more recent methods aim to address this issue. The present work scrutinizes the most popular metacognitive performance measures in terms of two critical characteristics: independence of type 1 performance and test-retest reliability. Analyses of data from the Confidence Database (total N = 6912) indicate that no current metacognitive performance measure is independent of type 1 performance. The shape of this dependency is largely reproduced by extending current models of metacognition with a source of metacognitive noise. Moreover, the reliability of metacognitive performance measures is highly sensitive to the combination of type 1 performance and trial number. Importantly, trial numbers frequently employed in metacognition research are too low to achieve an acceptable level of test-retest reliability. Among common task characteristics, simultaneous choice and confidence reports most strongly improved reliability. Finally, general recommendations about design choices and analytical remedies for studies investigating metacognitive performance are provided.

scholarly journals Validity and reliability of metacognitive performance measures

2021 ◽  
Author(s):  
Matthias Guggenmos
Keyword(s):  
Performance Measures ◽  
Performance Measure ◽  
Human Cognition ◽  
Validity And Reliability ◽  
Retest Reliability ◽  
Highly Sensitive ◽  
Primary Type ◽  
Test Retest Reliability ◽  
Sensory Noise

Research on metacognition − thinking about thinking − has grown rapidly and fostered our understanding of human cognition in healthy individuals and clinical populations. Of central importance is the concept of metacognitive performance, which characterizes the capacity of an individual to estimate and report the accuracy of primary (type 1) cognitive processes or actions ensuing from these processes. Arguably one of the biggest challenges for measures of metacognitive performance is their dependency on objective type 1 performance, although more recent methods aim to address this issue. In the present work we scrutinize the most popular metacognitive performance measures in terms of their validity (independence of type 1 performance) and test-retest reliability. Based on data of the Confidence Database we find that no current metacognitive performance measure is independent of type 1 performance. The shape of this dependency is reproduced by a simple computational model which considers metacognitive noise in addition to sensory noise. Moreover, we show that the reliability of metacognitive performance measures is highly sensitive to the combination of type 1 performance and trial number. Critically, trial numbers frequently employed in metacognition research are too low to achieve an acceptable level of test-retest reliability. Finally, we investigate design choices and analytical remedies to improve both validity and reliability and provide general recommendations for studies investigating metacognitive performance.

Test-Retest Reliability of the Trauma and Life Events Self-Report Inventory

2000 ◽  
Vol 87 (3) ◽  
pp. 750-752 ◽  
Author(s):  
J. E. Hovens ◽  
I. Bramsen ◽  
H. M. van der Ploeg ◽  
I. E. W. Reuling
Keyword(s):  
Medical Students ◽  
Life Events ◽  
Self Report ◽  
First Year ◽  
Total N ◽  
Male And Female ◽  
Retest Reliability ◽  
Time Periods ◽  
Test Retest Reliability

Three groups of first-year male and female medical students (total N = 90) completed the Trauma and Life Events Self-report Inventory twice. Test-retest reliability for the three different time periods was .82, .89, and .75, respectively.

scholarly journals The Development and Psychometric Validation of the Diabetes Impact and Device Satisfaction Scale for Individuals with Type 1 Diabetes

2020 ◽  
Vol 14 (2) ◽  
pp. 309-317
Author(s):  
Michelle L. Manning ◽  
Harsimran Singh ◽  
Keaton Stoner ◽  
Steph Habif
Keyword(s):  
Type 1 Diabetes ◽  
Assessment Tool ◽  
Rapid Development ◽  
Insulin Delivery ◽  
Psychometric Validation ◽  
Specific Information ◽  
Retest Reliability ◽  
Test Retest Reliability ◽  
Post Assessment

Background: With the rapid development of new insulin delivery technology, measuring patient experience has become especially pertinent. The current study reports on item development, psychometric validation, and intended use of the newly developed Diabetes Impact and Device Satisfaction (DIDS) Scale. Method: The DIDS Scale was informed by a comprehensive literature review, and field tested as part of two focus groups. The finalized measure was used at baseline and 6 months post-assessment with a large US cohort. Exploratory factor analyses (EFAs) were conducted to determine and confirm factor structure and item selection. Internal reliability, test–retest reliability, and convergent/divergent validity of the emerged factors were tested with demographics, diabetes-specific information, and diabetes behavioral and satisfaction measures. Results: In all, 778 participants with type 1 diabetes (66% female, mean age 47.13 ± 17.76 years, 74% insulin pump users) completed surveys at both baseline and post-assessment. EFA highlighted two factors—Device Satisfaction (seven items, Cronbach’s α = 0.85-0.90) and Diabetes Impact (four items, Cronbach’s α = 0.71-0.75). DIDS Scale demonstrated good concurrent validity and test–retest reliability. Conclusion: The DIDS Scale is a novel and a brief assessment tool with robust psychometric properties. It is recommended for use across all insulin delivery devices and is considered appropriate for use in longitudinal studies. Future studies are recommended to evaluate the performance of DIDS Scale in diverse populations with diabetes.

The test–retest reliability of motor performance measures after traumatic brain injury

2006 ◽  
Vol 8 (2) ◽  
pp. 50-59 ◽  
Author(s):  
Matti V. Vartiainen ◽  
Marjo B. Rinne ◽  
Tommi M. Lehto ◽  
Matti E. Pasanen ◽  
Jaana M. Sarajuuri ◽  
...  
Keyword(s):  
Traumatic Brain Injury ◽  
Brain Injury ◽  
Performance Measures ◽  
Motor Performance ◽  
Retest Reliability ◽  
Test Retest Reliability

scholarly journals Test–retest reliability of self-reported diabetes diagnosis in the Norwegian Women and Cancer Study: A population-based longitudinal study (n =33,919)

2016 ◽  
Vol 4 ◽  
pp. 205031211562285 ◽  
Author(s):  
Mashhood Ahmed Sheikh ◽  
Eiliv Lund ◽  
Tonje Braaten
Keyword(s):  
Physical Activity ◽  
Type 2 Diabetes ◽  
Diabetes Diagnosis ◽  
Clear Pattern ◽  
Retest Reliability ◽  
Cancer Study ◽  
Test Retest Reliability ◽  
Kappa Agreement

Objective: Self-reported information from questionnaires is frequently used in epidemiological studies, but few of these studies provide information on the reproducibility of individual items contained in the questionnaire. We studied the test–retest reliability of self-reported diabetes among 33,919 participants in Norwegian Women and Cancer Study. Methods: The test–retest reliability of self-reported type 1 and type 2 diabetes diagnoses was evaluated between three self-administered questionnaires (completed in 1991, 1998, and 2005 by Norwegian Women and Cancer participants) by kappa agreement. The time interval between the test–retest studies was ~7 and ~14 years. Sensitivity of the kappa agreement for type 1 and type 2 diabetes diagnoses was assessed. Subgroup analysis was performed to assess whether test–retest reliability varies with age, body mass index, physical activity, education, and smoking status. Results: The kappa agreement for both types of self-reported diabetes diagnoses combined was good (⩾0.65) for all three test–retest studies (1991–1998, 1991–2005, and 1998–2005). The kappa agreement for type 1 diabetes was good (⩾0.73) in the 1991–2005 and the 1998–2005 test–retest studies, and very good (0.83) in the 1991–1998 test–retest study. The kappa agreement for type 2 diabetes was moderate (0.57) in the 1991–2005 test–retest study and good (⩾0.66) in the 1991–1998 and 1998–2005 test–retest studies. The overall kappa agreement in the 1991–1998 test–retest study was stronger than in the 1991–2005 test–retest study and the 1998–2005 test–retest study. There was no clear pattern of inconsistency in the kappa agreements within different strata of age, BMI, physical activity, and smoking. The kappa agreement was strongest among the respondents with 17 or more years of education, while generally it was weaker among the least educated group. Conclusion: The test–retest reliability of the diabetes was acceptable and there was no clear pattern of inconsistency in the kappa agreement stratified by age, body mass index, physical activity, and smoking. The study suggests that self-reported diabetes diagnosis from middle-aged women enrolled in the Norwegian Women and Cancer Study is reliable.

Sign in / Sign up

Export Citation Format

Share Document