Validation for Performance Measurement in the Task of Fault Management

1983 ◽  
Vol 27 (13) ◽  
pp. 1058-1062
Author(s):  
Brian W. Surgenor ◽  
John D. McGeachy
Keyword(s):  
Performance Measurement ◽  
External Validity ◽  
Human Performance ◽  
External Validation ◽  
Fault Management ◽  
Internal Validity ◽  
Essential Elements ◽  
Internal Validation ◽  
Fundamental Understanding ◽  
Internal And External Validity

The application of a part-task nuclear simulator to the measurement of performance in the task of fault management was studied. Specifically, the design of the simulator was evaluated for internal and external validity. External validation required confirmation that the task presented by the simulator had the essential elements of the real task. In the context of this particular study, internal validation required confirmation that performance on the simulator represented an accurate and fair measure of a subject's understanding of fundamental principles. The requirements for external and internal validity were found to be in conflict. Performance on the simulator was not an accurate measure of fundamental understanding because the task was realistic. However, it was concluded that a part-task simulator does provide an effective method of gathering information on human performance.

scholarly journals A novel preference-informed complementary trial (PICT) design for clinical trial research influenced by strong patient preferences

Trials ◽  
2021 ◽  
Vol 22 (1) ◽  
Author(s):  
Samina Ali ◽  
◽  
Gareth Hopkin ◽  
Naveen Poonai ◽  
Lawrence Richer ◽  
...  
Keyword(s):  
External Validity ◽  
Treatment Options ◽  
Internal Validity ◽  
Clinical Effectiveness ◽  
Drop Out ◽  
Innovative Design ◽  
Clinical Trial Research ◽  
Research Questions ◽  
Internal And External Validity ◽  
Treatment Population

Abstract Background Patients and their families often have preferences for medical care that relate to wider considerations beyond the clinical effectiveness of the proposed interventions. Traditionally, these preferences have not been adequately considered in research. Research questions where patients and families have strong preferences may not be appropriate for traditional randomized controlled trials (RCTs) due to threats to internal and external validity, as there may be high levels of drop-out and non-adherence or recruitment of a sample that is not representative of the treatment population. Several preference-informed designs have been developed to address problems with traditional RCTs, but these designs have their own limitations and may not be suitable for many research questions where strong preferences and opinions are present. Methods In this paper, we propose a novel and innovative preference-informed complementary trial (PICT) design which addresses key weaknesses with both traditional RCTs and available preference-informed designs. In the PICT design, complementary trials would be operated within a single study, and patients and/or families would be given the opportunity to choose between a trial with all treatment options available and a trial with treatment options that exclude the option which is subject to strong preferences. This approach would allow those with strong preferences to take part in research and would improve external validity through recruiting more representative populations and internal validity. Here we discuss the strengths and limitations of the PICT design and considerations for analysis and present a motivating example for the design based on the use of opioids for pain management for children with musculoskeletal injuries. Conclusions PICTs provide a novel and innovative design for clinical trials with more than two arms, which can address problems with existing preference-informed trial designs and enhance the ability of researchers to reflect shared decision-making in research as well as improving the validity of trials of topics with strong preferences.

scholarly journals High Fidelity Microsurgical Simulation: The Thiel Model and Evaluation Instrument

2018 ◽  
Vol 27 (2) ◽  
pp. 118-124 ◽  
Author(s):  
Andrei Odobescu ◽  
Isak Goodwin ◽  
Djamal Berbiche ◽  
Joseph BouMerhi ◽  
Patrick G. Harris ◽  
...  
Keyword(s):  
External Validity ◽  
Internal Validity ◽  
High Fidelity ◽  
Evaluation Instrument ◽  
Intra Class Correlation ◽  
Validity Measures ◽  
The Difference ◽  
Fidelity Model ◽  
Junior Residents ◽  
Internal And External Validity

Background: The Thiel embalmment method has recently been used in a number of medical simulation fields. The authors investigate the use of Thiel vessels as a high fidelity model for microvascular simulation and propose a new checklist-based evaluation instrument for microsurgical training. Methods: Thirteen residents and 2 attending microsurgeons performed video recorded microvascular anastomoses on Thiel embalmed arteries that were evaluated using a new evaluation instrument (Microvascular Evaluation Scale) by 4 fellowship trained microsurgeons. The internal validity was assessed using the Cronbach coefficient. The external validity was verified using regression models. Results: The reliability assessment revealed an excellent intra-class correlation of 0.89. When comparing scores obtained by participants from different levels of training, attending surgeons and senior residents (Post Graduate Year [PGY] 4-5) scored significantly better than junior residents (PGY 1-3). The difference between senior residents and attending surgeons was not significant. When considering microsurgical experience, the differences were significant between the advanced group and the minimal and moderate experience groups. The differences between minimal and moderate experience groups were not significant. Based on the data obtained, a score of 8 would translate into a level of microsurgical competence appropriate for clinical microsurgery. Conclusions: Thiel cadaveric vessels are a high fidelity model for microsurgical simulation. Excellent internal and external validity measures were obtained using the Microvascular Evaluation Scale (MVES).

scholarly journals Using the RE-AIM framework to evaluate internal and external validity of mobile phone–based interventions in diabetes self-management education and support

2020 ◽  
Vol 27 (6) ◽  
pp. 946-956 ◽  
Author(s):  
Yilin Yoshida ◽  
Sonal J Patil ◽  
Ross C Brownson ◽  
Suzanne A Boren ◽  
Min Kim ◽  
...  
Keyword(s):  
External Validity ◽  
Management Education ◽  
Data Extraction ◽  
Internal Validity ◽  
Self Management ◽  
Future Research ◽  
Limited Information ◽  
Quality Of Reporting ◽  
Screening Process ◽  
Internal And External Validity

Abstract Objective We evaluated the extent to which studies that tested short message service (SMS)– and application (app)-based interventions for diabetes self-management education and support (DSMES) report on factors that inform both internal and external validity as measured by the RE-AIM (Reach, Efficacy/Effectiveness, Adoption, Implementation, and Maintenance) framework. Materials and Methods We systematically searched PubMed, Embase, Web of Science, CINAHL (Cumulative Index of Nursing and Allied Health Literature), and IEEE Xplore Digital Library for articles from January 1, 2009, to February 28, 2019. We carried out a multistage screening process followed by email communications with study authors for missing or discrepant information. Two independent coders coded eligible articles using a 23-item validated data extraction tool based on the RE-AIM framework. Results Twenty studies (21 articles) were included in the analysis. The comprehensiveness of reporting on the RE-AIM criteria across the SMS- and app-based DSMES studies was low. With respect to internal validity, most interventions were well described and primary clinical or behavioral outcomes were measured and reported. However, gaps exist in areas of attrition, measures of potential negative outcomes, the extent to which the protocol was delivered as intended, and description on delivery agents. Likewise, we found limited information on external validity indicators across adoption, implementation, and maintenance domains. Conclusions Reporting gaps were found in internal validity but more so in external validity in the current SMS- and app-based DSMES literature. Because most studies in this review were efficacy studies, the generalizability of these interventions cannot be determined. Future research should adopt the RE-AIM dimensions to improve the quality of reporting and enhance the likelihood of translating research to practice.

Validating vascular access data in the Swedish Renal Registry SRR

2020 ◽  
pp. 112972982095473
Author(s):  
Gunilla Welander ◽  
Birgitta Sigvant
Keyword(s):  
Vascular Access ◽  
Medical Records ◽  
External Validation ◽  
Internal Validity ◽  
Validation Data ◽  
Internal Validation ◽  
Clinical Utilization ◽  
Surgical Units ◽  
National Patient ◽  
Access Data

Background: All Swedish dialysis units register data on vascular access in the Swedish Renal Registry (SRR). This study assessed external and internal validity of vascular access data in the SRR and its use as a tool in clinical practice. Methods: For external validation, all procedures for placed fistulas, open and endovascular reinterventions registered in the SRR in 2011 to 2017 were cross-matched with data from the Swedish National Patient Registry. A two-stage sampling selected 12/60 dialysis units for internal validation. Data on current vascular access for 10 randomly selected patients at each unit were compared with medical record data. SRR data on placed fistulas from 2017 were cross-checked with data from local surgical units. Registrations of central venous catheters (CVCs) as temporary or permanent were used as a proxy for clinical utilization of the registry and analyzed separately. Results: External validity increased from 74% to 83% during the observation period. In all, 1037 datapoints were used in internal validation, with a 95% match between SRR registrations and medical records. Registrations of CVCs, fistulas, and interventions were reliable, with few missing data or mismatches. Vascular access type initiating hemodialysis was missing or incorrect in either the SRR or medical records for 14/120 patients. Registrations of placed fistulas in 2017 matched in all but four (pre-dialysis stage) of 135 cases. Some 35% of the CVCs validated ( n = 49) at 7/12 units were not categorized as temporary or permanent. Conclusion: The SRR provides a reliable resource on current vascular access care.

scholarly journals A New Predictive Model for Breast Cancer Survival in New Zealand: Development, Internal and External Validation, and Comparison With the Nottingham Prognostic Index

2018 ◽  
Vol 4 (Supplement 2) ◽  
pp. 227s-227s
Author(s):  
M. Elwood ◽  
S. Tin Tin ◽  
E. Tawfiq ◽  
R.J. Marshall ◽  
T.M. Phung ◽  
...  
Keyword(s):  
Breast Cancer ◽  
New Zealand ◽  
External Validity ◽  
External Validation ◽  
Prognostic Index ◽  
Population Based ◽  
Nottingham Prognostic Index ◽  
Data Set ◽  
Kaplan Meier ◽  
Internal And External Validity

Background: Women diagnosed with breast cancer, their doctors, and their families, would find a valid estimate of her prognosis helpful in planning treatment and support. Assessing prognosis is complex as many factors influence it. Several predictive models have been produced, but none has been developed or tested on patients in New Zealand (NZ). Aim: We aimed to develop and validate a NZ predictive model (NZPM) for breast cancer, and compare its performance to a widely used UK-developed model, the Nottingham Prognostic Index (NPI). Methods: We developed a model to predict 10-year breast cancer-specific survival, using data collected prospectively in the largest population-based breast cancer registry in NZ (Auckland, 9182 patients), and assessed its performance in this data set (internal validation) and in an independent NZ population-based series of 2625 patients in Waikato (external validation). The data included all women with primary invasive breast cancer diagnosed from 1 June 2000 to 30 June 2014, with follow-up to death or to 31 December 2014. We used multivariate Cox proportional hazards regression to assess predictors and to estimate the probability of breast cancer mortality within 10 years, and therefore 10-year survival, for each patient. We assessed observed survival by the Kaplan-Meier method. We assessed discrimination by the C-statistic, and calibration by comparing predicted and observed survival rates for patients in 10 groups ordered by predicted 10-year survival. We compared this NZPM with the NPI in the validation data set. Results: The final NZPM used continuous variables of age, tumor size, and number of positive lymph nodes, and categorical variables of ethnicity, tumor stage, tumor grade, ER and PR receptors, HER2 status, and histologic type of tumor. Discrimination was good: C-statistics were 0.84 for internal validity and 0.83 for independent external validity. For calibration, for both internal and external validity, the predicted 10-year survival probabilities in 10 groups of patients, ordered by predicted survival, were all within the 95% confidence intervals (CI) of the observed Kaplan-Meier survival probabilities. The NZPM showed good discrimination even within the prognostic groups defined by the NPI. Conclusion: These results for the NZPM show good internal and external validity, transportability, potential clinical value, and its clear superiority over the NPI. Further research will assess other potential predictors, other outcomes, performance in specific subgroups of patients, and compare the NZPM to other models, which have been developed in other countries and have not yet been tested in NZ.

Internal and External Validity in Ethical Reasoning

2021 ◽  
pp. 40-61
Author(s):  
James Wilson
Keyword(s):  
Real World ◽  
External Validity ◽  
Thought Experiment ◽  
Ethical Reasoning ◽  
Internal Validity ◽  
Thought Experiments ◽  
American Philosophy ◽  
Anglo American ◽  
Internal And External Validity ◽  
Rigorous Method

A particular approach to ethical reasoning has come to dominate much Anglo-American philosophy, one which assumes that the most rigorous method is to proceed by analysis of thought experiments. In thought experiments, features such as context and history are stripped away, and all factors other than those of ethical interest are stipulated to be equal. This chapter argues that even if a thought experiment produces results that are internally valid—in that it provides a genuine ethical insight about the highly controlled and simplified experimental scenario under discussion—this does not imply external validity. Just as in empirical experiments, there is a yawning gap between succeeding in the relatively easy project of establishing internal validity in a controlled and simplified context, and the more difficult one of establishing external validity in the messier and more complex real world.

The internal and external validity of the Major Depression Inventory in measuring severity of depressive states

2003 ◽  
Vol 33 (2) ◽  
pp. 351-356 ◽  
Author(s):  
L. R. OLSEN ◽  
D. V. JENSEN ◽  
V. NOERHOLM ◽  
K. MARTINY ◽  
P. BECH
Keyword(s):  
Major Depression ◽  
External Validity ◽  
Depression Scale ◽  
Internal Validity ◽  
Depressive Illness ◽  
Hamilton Depression Scale ◽  
Major Depression Inventory ◽  
Icd 10 ◽  
Internal And External Validity ◽  
Depressive States

Background. We have developed the Major Depression Inventory (MDI), consisting of 10 items, covering the DSM-IV as well as the ICD-10 symptoms of depressive illness. We aimed to evaluate this as a scale measuring severity of depressive states with reference to both internal and external validity.Method. Patients representing the score range from no depression to marked depression on the Hamilton Depression Scale (HAM-D) completed the MDI. Both classical and modern psychometric methods were applied for the evaluation of validity, including the Rasch analysis.Results. In total, 91 patients were included. The results showed that the MDI had an adequate internal validity in being a unidimensional scale (the total score an appropriate or sufficient statistic). The external validity of the MDI was also confirmed as the total score of the MDI correlated significantly with the HAM-D (Pearson's coefficient 0·86, P[les ]0·01, Spearman 0·80, P[les ]0·01).Conclusion. When used in a sample of patients with different states of depression the MDI has an adequate internal and external validity.

scholarly journals Experimental and Non-Experimental Methods in Development Economics: A Porous Dialectic

2015 ◽  
Vol 6 (1) ◽  
Author(s):  
Rajeev Dehejia
Keyword(s):  
Randomized Controlled Trials ◽  
External Validity ◽  
Internal Validity ◽  
Experimental Methods ◽  
Controlled Trials ◽  
Data Sets ◽  
Randomized Controlled ◽  
Nationally Representative ◽  
High Degree ◽  
Internal And External Validity

AbstractThis paper surveys six widely-used non-experimental methods for estimating treatment effects (instrumental variables, regression discontinuity, direct matching, propensity score matching, linear regression and non-parametric methods, and difference-in-differences), and assesses their internal and external validity relative both to each other and to randomized controlled trials. While randomized controlled trials can achieve the highest degree of internal validity when cleanly implemented in the field, the availability of large, nationally representative data sets offers the opportunity for a high degree of external validity using non-experimental methods. We argue that each method has merits in some context and they are complements rather than substitutes.

Replicating Experiments Using Aggregate and Survey Data: The Case of Negative Advertising and Turnout

1999 ◽  
Vol 93 (4) ◽  
pp. 901-909 ◽  
Author(s):  
Stephen D. Ansolabehere ◽  
Shanto Iyengar ◽  
Adam Simon
Keyword(s):  
Survey Data ◽  
Instrumental Variables ◽  
External Validity ◽  
External Validation ◽  
Internal Validity ◽  
Aggregate Data ◽  
Causal Effects ◽  
Negative Advertising ◽  
Senate Elections ◽  
Advertising Exposure

Experiments show significant demobilizing and alienating effects of negative advertising. Although internally valid, experiments may have limited external validity. Aggregate and survey data offer two ways of providing external validation for experiments. We show that survey recall measures of advertising exposure suffer from problems of internal validity due to simultaneity and measurement error, which bias estimated effects of ad exposure. We provide valid estimates of the causal effects of ad exposure for the NES surveys using instrumental variables and find that negative advertising causes lower turnout in the NES data. We also provide a careful statistical analysis of aggregate turnout data from the 1992 Senate elections that Wattenberg and Brians (1999) recommend. These aggregate data confirm our original findings. Experiments, surveys, and aggregate data all point to the same conclusion: Negative advertising demobilizes voters.

scholarly journals Who’s in and Who’s Out? Selection Bias in Aging Research

2020 ◽  
Vol 4 (Supplement_1) ◽  
pp. 822-822
Author(s):  
Elizabeth Rose Mayeda ◽  
Eleanor Hayes-Larson ◽  
Hailey Banack
Keyword(s):  
Selection Bias ◽  
External Validity ◽  
Sample Selection ◽  
Internal Validity ◽  
Causal Effects ◽  
Aging Research ◽  
Selection Processes ◽  
The People ◽  
Biased Estimates ◽  
Internal And External Validity

Abstract Selection bias presents a major threat to both internal and external validity in aging research. “Selection bias” refers to sample selection processes that lead to statistical associations in the study sample that are biased estimates of causal effects in the population of interest. These processes can lead to: (1) results that do not generalize to the population of interest (threat to external validity) or (2) biased effect estimates (associations that do not represent causal effects for any population, including the people in the sample; a threat to internal validity). In this presentation, we give an overview of selection bias in aging research. We will describe processes that can give rise to selection bias, highlight why they are particularly pervasive in this field, and present several examples of selection bias in aging research. We end with a brief summary of strategies to prevent and correct for selection bias in aging research.

Sign in / Sign up

Export Citation Format

Share Document