scholarly journals Direct observation of procedural skills (DOPS) assessment in diagnostic gastroscopy: nationwide evidence of validity and competency development during training

2019 ◽  
Vol 34 (1) ◽  
pp. 105-114 ◽  
Author(s):  
Keith Siau ◽  
◽  
James Crossley ◽  
Paul Dunckley ◽  
Gavin Johnson ◽  
...  
Keyword(s):  
Factor Analysis ◽  
Exploratory Factor Analysis ◽  
Direct Observation ◽  
Roc Analysis ◽  
Assessment Tool ◽  
Technical Skills ◽  
Competency Assessment ◽  
Competency Development ◽  
Procedural Skills ◽  
Post Procedure

AbstractBackgroundValidated competency assessment tools and the data supporting milestone development during gastroscopy training are lacking. We aimed to assess the validity of the formative direct observation of procedural skills (DOPS) assessment tool in diagnostic gastroscopy and study competency development using DOPS.MethodsThis was a prospective multicentre (N = 275) analysis of formative gastroscopy DOPS assessments. Internal structure validity was tested using exploratory factor analysis and reliability estimated using generalisability theory. Item and global DOPS scores were stratified by lifetime procedure count to define learning curves, using a threshold determined from receiver operator characteristics (ROC) analysis. Multivariable binary logistic regression analysis was performed to identify independent predictors of DOPS competence.ResultsIn total, 10086 DOPS were submitted for 987 trainees. Exploratory factor analysis identified three distinct item groupings, representing ‘pre-procedure’, ‘technical’, and ‘post-procedure non-technical’ skills. From generalisability analyses, sources of variance in overall DOPS scores included trainee ability (31%), assessor stringency (8%), assessor subjectivity (18%), and trainee case-to-case variation (43%). The combination of three assessments from three assessors was sufficient to achieve the reliability threshold of 0.70. On ROC analysis, a mean score of 3.9 provided optimal sensitivity and specificity for determining competency. This threshold was attained in the order of ‘pre-procedure’ (100–124 procedures), ‘technical’ (150–174 procedures), ‘post-procedure non-technical’ skills (200–224 procedures), and global competency (225–249 procedures). Higher lifetime procedure count, DOPS count, surgical trainees and assessors, higher trainee seniority, and lower case difficulty were significant multivariable predictors of DOPS competence.ConclusionThis study establishes milestones for competency acquisition during gastroscopy training and provides validity and reliability evidence to support gastroscopy DOPS as a competency assessment tool.

scholarly journals Colonoscopy Direct Observation of Procedural Skills Assessment Tool for Evaluating Competency Development During Training

2019 ◽  
Vol 115 (2) ◽  
pp. 234-243 ◽  
Author(s):  
Keith Siau ◽  
James Crossley ◽  
Paul Dunckley ◽  
Gavin Johnson ◽  
Mark Feeney ◽  
...  
Keyword(s):  
Direct Observation ◽  
Assessment Tool ◽  
Skills Assessment ◽  
Competency Development ◽  
Procedural Skills

scholarly journals Training and Assessment in Flexible Sigmoidoscopy: using a Novel Direct Observation of Procedural Skills (DOPS) Assessment Tool

2019 ◽  
Vol 28 (1) ◽  
pp. 33-40
Author(s):  
Keith Siau ◽  
James Crossley ◽  
Paul Dunckley ◽  
Gavin Johnson ◽  
Adam Haycock ◽  
...  
Keyword(s):  
Direct Observation ◽  
Flexible Sigmoidoscopy ◽  
Assessment Tool ◽  
Binary Logistic Regression ◽  
Learning Curves ◽  
Competency Development ◽  
Theory Analysis ◽  
Procedural Skills ◽  
Reliability Estimates ◽  
National Training

Background & Aims: Data supporting milestone development during flexible sigmoidoscopy (FS) training are lacking. We aimed to present validity evidence for our formative direct observation of procedural skills (DOPS) assessment in FS, and use DOPS to establish competency benchmarks and define learning curves for a national training cohort.Methods: This prospective UK-wide (211 centres) study included all FS formative DOPS assessments submitted to the national e-portfolio. Reliability was estimated from generalisability theory analysis. Item and global DOPS scores were correlated with lifetime procedure count to study learning curves, with competency benchmarks defined using contrasting groups analysis. Multivariable binary logistic regression was performedto identify independent predictors of DOPS competence.Results: This analysis included 3,616 DOPS submitted for 468 trainees. From generalisability analysis, sources of overall competency score variance included: trainee ability (27%), assessor stringency (15%), assessor subjectivity attributable to the trainee (18%) and case-to-case variation (40%), which enabled the modelling of reliability estimates. The competency benchmark (mean DOPS score: 3.84) was achieved after 150-174 procedures. Across the cohort, competency development occurred in the order of: pre-procedural (50-74), non-technical (75-149), technical (125-174) and post-procedural (175-199) skills. Lifetime procedural count (p<0.001), case difficulty (p<0.001), and lifetime formative DOPS count (p=0.001) were independently associated with DOPS competence, but not trainee or assessor specialty.Conclusion: Sigmoidoscopy DOPS can provide valid and reliable assessments of competency during training and can be used to chart competency development. Contrary to earlier studies, based on destination-orientated endpoints, overall competency in sigmoidoscopy was attained after 150 lifetime procedures.

scholarly journals A Call for Leadership and Management Competency Development for Directors of Medical Services—Evidence from the Chinese Public Hospital System

2020 ◽  
Vol 17 (18) ◽  
pp. 6913
Author(s):  
Zhanming Liang ◽  
Peter Howard ◽  
Jian Wang ◽  
Min Xu
Keyword(s):  
Assessment Tool ◽  
System Level ◽  
Medical Services ◽  
Competency Assessment ◽  
Competency Development ◽  
Hospital System ◽  
Health Service Provision ◽  
Leadership And Management ◽  
Medical Leadership ◽  
Management Competency

Background: A competent medical leadership and management workforce is key to the effectiveness and efficiency of health service provision and to leading and managing the health system reform agenda in China. However, the traditional recruitment and promotion approach of relying on clinical performance and seniority provides limited incentive for competency development and improvement. Methods: A three-component survey including the use of a validated management competency assessment tool was conducted with Directors of Medical Services (n = 143) and Deputy Directors of Medical Services (n = 152) from three categories of hospital in Jinan, Shandong Province, China. Results: The survey identified the inadequacy of formal and informal management training received by hospital medical leaders before commencing their management positions and confirms that the low self-perceived competency level across two medical management level and three hospitals was beyond acceptable. The study also indicates that the informal and formal education provided to Chinese medical leaders have not been effective in developing the required management competencies. Conclusions: The study suggests two system level approaches (health and higher education systems) and one organization level approach to formulate overall medical leadership and management workforce development strategies to encourages continuous management competency development and self-improvement among clinical leaders in China.

scholarly journals Development of Emergency Response Team Technical Competency (ER2TC) Assessment Tool for Offshore Process Safety

2021 ◽  
Vol 4 (1) ◽  
Author(s):  
Wen Khai Lim ◽  
Risza Rusli ◽  
Azizul Buang ◽  
Taram Satiraksa Wan Abdullah
Keyword(s):  
Emergency Response ◽  
Assessment Tool ◽  
Technical Skills ◽  
Oil And Gas Industry ◽  
Competency Assessment ◽  
Safety Critical ◽  
Technical Competency ◽  
Response Team ◽  
Offshore Industry ◽  
The Impact

The importance in identifying safety critical task in high-risk industries for competence assessment is greatly emphasised to ensure the personnel is well equipped with high level of competence and assurance in certain safety critical tasks. Lack of competency had resulted in various accidents in oil and gas industry. Failure in escape, evacuation, and rescue (EER) operation in history has led to tragic consequences associated with high number of fatalities, such as Alexander L. Kielland platform collapse, Piper Alpha disaster and tragedy of Ocean Ranger. Although competency assessment is widely implemented in offshore industry, accident still occur which indicates that reliable competency assessment is highly essential. In this research, a technical competency assessment tool is developed to assess the technical skills of each individual in emergency response team during EER activities. Case studies are selected to evaluate the designed Emergency Response Team Technical Competency (ER2TC) assessment tool where a range of different inputs and parameters are designed and tested on the designed tool. Analysis is conducted to identify how the overall output is affected by the uncertainty from the designed tool’s input and discover the impact of potential errors in the tool towards the output generated from the tool. The ER2TC assessment tool overcomes personal subjectivity and biasness of assessors, thus, producing reusable and reliable tool for decision makers in the evaluation of candidates. This tool has also implemented stricter assessing criteria to evaluate the performance of candidates in EER activities. This is seen as necessary due to the critical nature and the must to ensure the successful for EER operations as any failure could potentially results in loss of lives. Thus, this ER2TC assessment tool has indeed sufficient to assess the technical skills of personnel in ensuring the success of EER operations in offshore installations.

scholarly journals Non-technical skills assessment scale in nursing: construction, development and validation

2018 ◽  
Vol 26 (0) ◽  
Author(s):  
Sara Martins Pereira Pires ◽  
Sara Otília Marques Monteiro ◽  
Anabela Maria Sousa Pereira ◽  
Joana Novaes Machado Stocker ◽  
Daniela de Mascarenhas Chaló ◽  
...  
Keyword(s):  
Factor Analysis ◽  
Nursing Education ◽  
Exploratory Factor Analysis ◽  
Undergraduate Students ◽  
Technical Skills ◽  
Skills Training ◽  
Assessment Scale ◽  
Skills Assessment ◽  
Response Options ◽  
Technical Skills Assessment

ABSTRACT The introduction of non-technical skills during nursing education is crucial to prepare nurses for the clinical context and increase patient safety. We found no instrument developed for this purpose. Objectives: to construct, develop and validate a non-technical skills assessment scale in nursing. Method: methodological research. Based on the literature review and experience of researchers on non-technical skills in healthcare and the knowledge of the principles of crisis resource management, a list of 63 items with a five-point Likert scale was constructed. The scale was applied to 177 nursing undergraduate students. Descriptive statistics, correlations, internal consistency analysis and exploratory factor analysis were performed to evaluate the psychometric properties of the scale. Results: scale items presented similar values for mean and median. The maximum and the minimum values presented a good distribution amongst all response options. Most items presented a significant and positive relationship. Cronbach alpha presented a good value (0.94), and most correlations were significant and positive. Exploratory factor analysis using the Kaiser-Meyer-Olkin test showed a value of 0.849, and the Bartlett’s test showed adequate sphericity values (χ2=6483.998; p=0.000). One-factor model explained 26% of the total variance. Conclusion: non-technical skills training and its measurement could be included in undergraduate or postgraduate courses in healthcare professions, or even be used to ascertain needs and improvements in healthcare contexts.

scholarly journals Development and Evaluation of the Validity and Reliability of the Leading and Managing Care Pre-Registration Nursing Student Assessment Tool

2021 ◽  
Vol 7 ◽  
pp. 237796082110002
Author(s):  
John Unsworth ◽  
Andrew Melling ◽  
Debra Porteous
Keyword(s):  
Factor Analysis ◽  
Internal Consistency ◽  
Assessment Tool ◽  
Nursing Leadership ◽  
Student Nurse ◽  
Competency Assessment ◽  
Theory Analysis ◽  
Leadership And Management ◽  
Clinical Nursing ◽  
Generalisability Theory

Background Clinical nursing leadership influences patient safety and the quality of care provided. Nurses at all levels require leadership and management skills. Despite recognition of the importance of leadership, student nurses often feel ill prepared to make the transition to Registered Nurse and struggle with prioritisation and delegation. In order to standardise student experience and promote the development of skills and attributes, a leadership and management competency assessment was developed and implemented. Aims This study aimed to identify the constructs that should be part of an assessment of student nurse competence in relation to clinical nursing leadership, and to evaluate the tool’s reliability. Method The first phase was to construct the competency assessment tool, using a mixture of deductive methods, including literature and expert review. Second, psychometric evaluation of the tool, including tests to examine its internal consistency and reliability, comparing test and retest reliability, exploratory factor analysis and generalisability theory analysis to identify reliability and sources of error. Results Five attributes were identified for inclusion in the tool alongside a scale of competence. 150 assessments were conducted with an average time between each assessment of three days. The results show that the tool was consistent over time with no significant difference in the mean scores. The Cronbach alpha was 0.84 and the tool had good internal consistency. The results of the factor analysis revealed loading onto a single construct. Generalisability theory analysis revealed 0.90 global reliability, with students accounting for the majority of the variation in scores. Conclusions The Leading and Managing Care assessment tool represents a valid and reliable assessment of student nurse competence to lead care delivery. Use of the tool during practice placement allows for a structured approach to the development of skills around prioritisation, management of resources, communication and the management of risk.

Sign in / Sign up

Export Citation Format

Share Document