Validity and reliability of an in-training evaluation report to measure the CanMEDS roles in emergency medicine residents

ABSTRACTBackground:There is a question of whether a single assessment tool can assess the key competencies of residents as mandated by the Royal College of Physicians and Surgeons of Canada CanMEDS roles framework.Objective:The objective of the present study was to investigate the reliability and validity of an emergency medicine (EM) in-training evaluation report (ITER).Method:ITER data from 2009 to 2011 were combined for residents across the 5 years of the EM residency training program. An exploratory factor analysis with varimax rotation was used to explore the construct validity of the ITER. A total of 172 ITERs were completed on residents across their first to fifth year of training.Results:A combined, 24-item ITER yielded a five-factor solution measuring the CanMEDs role Medical Expert/ Scholar, Communicator/Collaborator, Professional, Health Advocate and Manager subscales. The factor solution accounted for 79% of the variance, and reliability coefficients (Cronbach alpha) ranged from α = 0.90 to 0.95 for each subscale and α = 0.97 overall. The combined, 24-item ITER used to assess residents’ competencies in the EM residency program showed strong reliability and evidence of construct validity for assessment of the CanMEDS roles.Conclusions:Further research is needed to develop and test ITER items that will differentiate each CanMEDS role exclusively.

Download Full-text

Urdu Translation and Validation of Activities-Specific Balance Confidence (ABC) Scale

10.53350/pjmhs2115123505 ◽

2021 ◽

Vol 15 (12) ◽

pp. 3505-3508

Author(s):

Noor Ul Ain Fatima ◽

Qurat-Ul- Ain ◽

Fareeha Kausar ◽

Mian Ali Raza ◽

Misbah Waris ◽

...

Keyword(s):

Older Adults ◽

Construct Validity ◽

Correlation Coefficient ◽

Content Validity ◽

Assessment Tool ◽

Test Reliability ◽

Validity And Reliability ◽

A Value ◽

Abc Scale ◽

Translation And Validation

Objective: To translate and validate the ABC-Scale in Urdu language to predict risk of fall in older population. Study design: Cross-cultural Translation and validation Place and Duration: Study was conducted in older adult community of Sialkot from March 2020 to December 2020. Methodology: Translation of ABC in Urdu was conducted by using Beaton et al guidelines. Two bilingual translators translated the original version into Urdu language step wise, correction process was followed. Then two backward translations were done by language expert. After all this process, the translated version was reviewed by the professionals and the final version was applied on 15 individuals. Its reliability and validity was tested on 60 older adults. Results: For test re test reliability, intra class correlation coefficient ICC was measured with a value of 0.984 Which shows good test re-test reliability. The internal consistency and reliability of ABC was calculated by Cronbach’s alpha for total score with a value of 0.985. Content validity was good with values of CVI ranging from 0.767 to 0.955. To test the discriminative validity, independent t test was used to show the difference between the healthy and unhealthy adults. Factor analysis of UABC showed total variance 81.277 and cumulative variance was also 81.277. To calculate construct validity of U-ABC Pearson’s correlation coefficient was used and measured as 0.558. Conclusion: It was concluded that Urdu version of UABC is a valid assessment tool for older adults with fear of fall. It has good content validity, construct validity and reliability. Keywords: activities specific balance scale, validation, Urdu translation, reliability, tool translation

Download Full-text

Validity and reliability of Korean version Competency Assessment Tool-Mental Health

Frontiers of Nursing ◽

10.2478/fon-2020-0017 ◽

2020 ◽

Vol 7 (2) ◽

pp. 143-152

Author(s):

Hyun Mee Cho ◽

Jeong Won Han ◽

Eun Joung Choi ◽

Hyo Eun Jeong ◽

Bo Ram Hong ◽

...

Keyword(s):

Mental Health ◽

Health Care ◽

Mental Health Care ◽

Internal Consistency ◽

Assessment Tool ◽

Reliability And Validity ◽

Basic Data ◽

Measurement Tool ◽

Competency Assessment ◽

Validity And Reliability

AbstractObjectiveTo establish basic data for identifying the level of mental health care competency of general nurses through verification of the reliability and validity of Competency Assessment Tool-Mental Health in the development of a measurement tool for mental health care competency in Korea.MethodsThis study was conducted on nurses working at general wards, excluding those working at the Department of Psychiatry, in five hospitals with 200 beds or more located in Korea. Content, construct, concurrent validity, and internal consistency of the measurement were confirmed.ResultsAs a result of the construct validity, the section on importance of skills and knowledge for mental health care had 21 items, whereas the section on benefits of additional education had 22 items. The internal consistency of measurement was confirmed as follows: Cronbach's α = 0.96 for the section on importance and 0.96 for the section on benefits section.ConclusionsThis study verified the high validity and reliability of the tool in assessing the mental health care competency of nurses, and it is believed to be significant as basic data for enhancing such competency.

Download Full-text

Short Assessment of Health Literacy for Portuguese-speaking Adults

Revista de Saúde Pública ◽

10.1590/s0034-89102012005000047 ◽

2012 ◽

Vol 46 (4) ◽

pp. 702-711 ◽

Cited By ~ 48

Author(s):

Daniel Apolinario ◽

Rafaela de Castro Oliveira Pereira Braga ◽

Regina Miksian Magaldi ◽

Alexandre Leopold Busse ◽

Flavia Campora ◽

...

Keyword(s):

Older Adults ◽

Health Literacy ◽

Construct Validity ◽

Psychometric Properties ◽

Assessment Tool ◽

Characteristic Curve ◽

Intraclass Correlation ◽

Validity And Reliability ◽

Convenience Sample ◽

Inadequate Health Literacy

OBJECTIVE: To develop and validate a short health literacy assessment tool for Portuguese-speaking adults. METHODS: The Short Assessment of Health Literacy for Portuguese-speaking Adults is an assessment tool which consists of 50 items that assess an individual's ability to correctly pronounce and understand common medical terms. We evaluated the instrument's psychometric properties in a convenience sample of 226 Brazilian older adults. Construct validity was assessed by correlating the tool scores with years of schooling, self-reported literacy, and global cognitive functioning. Discrimination validity was assessed by testing the tool's accuracy in detecting inadequate health literacy, defined as failure to fully understand standard medical prescriptions. RESULTS: Moderate to high correlations were found in the assessment of construct validity (Spearman's coefficients ranging from 0.63 to 0.76). The instrument showed adequate internal consistency (Cronbach's alpha=0.93) and adequate test-retest reliability (intraclass correlation coefficient=0.95). The area under the receiver operating characteristic curve for detection of inadequate health literacy was 0.82. A version consisting of 18 items was tested and showed similar psychometric properties. CONCLUSIONS: The instrument developed showed good validity and reliability in a sample of Brazilian older adults. It can be used in research and clinical settings for screening inadequate health literacy.

Download Full-text

Reliability and Validity of Aids Victim Blaming Scales

Psychological Reports ◽

10.2466/pr0.1996.79.1.191 ◽

1996 ◽

Vol 79 (1) ◽

pp. 191-201 ◽

Cited By ~ 6

Author(s):

Charles L. Mulford ◽

Motoko Y. Lee

Keyword(s):

Construct Validity ◽

Sample Size ◽

Reliability And Validity ◽

Victim Blaming ◽

Validity And Reliability

Scales were developed to measure the tendencies to blame AIDS victims or the society in which victims live. Data from a sample of 874 students were used to examine the construct validity and reliability of the scales. The list-wise deletion reduced the sample size to the usable sample size of 824. The data showed satisfactory validity and reliability of the scales so they may be used to study attitudes toward those who are suffering from AIDS and related problems.

Download Full-text

Measurement of Multiple Intelligences among Sample of Students with Autism, and Intellectual Disability Using Teacher Estimation and Its Relationship with the Variables: The Type and Severity of Disability, Gender, Age, Type of Center

International Journal of Education ◽

10.5296/ije.v8i1.8268 ◽

2016 ◽

Vol 8 (1) ◽

pp. 107 ◽

Cited By ~ 2

Author(s):

Sabah Hasan Al-Onizat

Keyword(s):

Multiple Intelligences ◽

High Performance ◽

Assessment Tool ◽

Reliability And Validity ◽

Multiple Intelligence ◽

Mental Disability ◽

Validity And Reliability ◽

Musical Intelligence ◽

Severity Of Disability ◽

Students With Autism

This study aims to measure the multiple intelligence among sample of student with autism disorder and Mental disability by using teacher estimation in multiple intelligence scale and its relationship with the variables: type and severity of disability, gender, and the type of center, the study sample consisted of (81) student with autism and (85) student with mental disability, male and female, who enrolled in private and governmental situations in Amman. And for achieving the goals of the study the researcher developed the multiple intelligences assessment tool consists of (56) items, were verified validity and reliability of the tool where it was found that with acceptable degrees of reliability and validity. After processing the data statistically and analyzed, the results indicated that the most visible intelligence in mental disability is of musical intelligence, and in autism student is the kinesthetic intelligence, And Autistic children also showed superiority in arithmetic and kinesthetic intelligence compared with mental disability. Also children with mild disabilities have high performance in all type intelligences. And there are no differences in type of intelligence according to the gender and this proves that disability affects the brain regardless of the gender. While the adolescents have superiority in each of the linguistic, social and musical intelligence, there are no differences in the multiple intelligences depending on the type of center which the student attends in it.

Download Full-text

Application of Rasch to Verify Different Building Competency Sub Constructs for Competency Evaluation of Graduates in Nigeria

International Journal of Engineering and Advanced Technology - Regular Issue ◽

10.35940/ijeat.f1396.089620 ◽

2020 ◽

Vol 9 (6) ◽

pp. 253-260

Keyword(s):

Construct Validity ◽

Assessment Tool ◽

Survey Design ◽

Sampling Technique ◽

Survey Instrument ◽

Assessment Instrument ◽

Theoretical Interpretation ◽

Validity And Reliability ◽

Expert Opinions ◽

Competency Evaluation

Incessant concerns from employers and private sectors about the incompetence of graduates in Nigeria call for the creation of an assessment tool that could verify their skills. But there is no clear generally accepted and validated assessment instrument available for evaluating graduate performance. The aim of this study was to develop a valid instrument for assessing competency levels of building construction graduates in Nigeria. Survey design was adopted to obtain expert opinions on the validity of the sub-constructs and the related items about the employers 'needs. Three experts from the academic, public, and private sectors subjected the survey instrument to face, content, and construct validity and reliability. The survey instrument, which was analysed using IBM SPSS and WINSTEP version 3.73.3 was answered by a total of 200 building experts selected by proportionate stratified sampling technique. The consistency of the instrument was determined by fit statistics and point measure correlation (PTMEA Corr), for construct validity. The results revealed a very good items and person reliability of 0.97 and 0.94 respectively. Likewise, appropriate PTMEA Corr range from 0.36 to 0.68. Infit and outfit means square range obtained between 0.58 to 1.39. The findings give students, employers and academic institution a realistic and theoretical interpretation of the reality of labour market needs.

Download Full-text

Hot Seat Diagnosis: Competency-Based Tool Is Superior to Time-Based Tool for the Formative In-Service Assessment of Pathology Trainees

Archives of Pathology & Laboratory Medicine ◽

10.5858/arpa.2020-0702-ep ◽

2021 ◽

Author(s):

Rachel Han ◽

Julia Keith ◽

Elzbieta Slodkowska ◽

Sharon Nofech-Mozes ◽

Bojana Djordjevic ◽

...

Keyword(s):

Assessment Tool ◽

Training Evaluation ◽

Assessment Tools ◽

Assessment Scale ◽

Clinical Implications ◽

Competency Based Medical Education ◽

Evaluation Report ◽

Competency Based ◽

Diagnosis Tool ◽

Competency Based Assessment

Context.— Competency-based medical education relies on frequent formative in-service assessments to ascertain trainee progression. Currently at our institution, trainees receive a summative end-of-rotation In-Training Evaluation Report based on feedback collected from staff pathologists. There is no method of simulating report sign-out. Objective.— To develop a formative in-service assessment tool that is able to simulate report sign-out and provide case-by-case feedback to trainees. Further, to compare time- versus competency-based assessment models. Design.— Twenty-one pathology trainees were assessed for 20 months. Hot Seat Diagnosis by trainees and trainee assessment by pathologists were recorded in the Laboratory Information System. In the first iteration, trainees were assessed by using a time-based assessment scale on their ability to diagnose, report, use ancillary testings, comment on clinical implications, provide intraoperative consultation and/or gross cases. The second iteration used a competency-based assessment scale. Trainees and pathologists completed surveys on the effectiveness of the In-Training Evaluation Report versus the Hot Seat Diagnosis tool. Results.— Scores from both iterations correlated significantly with other assessment tools including the Resident In-Service Examination (r = 0.93, P = .04 and r = 0.87, P = .03). The competency-based model was better able to demonstrate improvement over time and stratify junior versus senior trainees than the time-based model. Trainees and pathologists rated Hot Seat Diagnosis as significantly more objective, detailed, and timely than the In-Training Evaluation Report, and effective at simulating report sign-out. Conclusions.— Hot Seat Diagnosis is an effective tool for the formative in-service assessment of pathology trainees and simulation of report sign-out, with the competency-based model outperforming the time-based model.

Download Full-text

Validity and Reliability Assessment Tool of Nursing Students Professional Behavior

Jurnal NERS ◽

10.20473/jn.v10i22015.208-216 ◽

2015 ◽

Vol 10 (2) ◽

pp. 208

Author(s):

Fatikhu Yatuni Asmara

Keyword(s):

Nursing Students ◽

Construct Validity ◽

Content Validity ◽

Assessment Tool ◽

Reliability Assessment ◽

Reliability Test ◽

Validity And Reliability ◽

Professional Behaviour ◽

Rater Reliability ◽

Validity Test

Introduction: Shieffield Peer Review Assessment (SPRAT) is an instrument to assess medical student’s performance using Multi Source Feedback (MSF) method. The previous study stated that MSF with SPRAT is an effective tool assessing professional behaviour (PB) of nursing students both in clinical setting and community setting. However it needs more items to be added. Based on that explanation, it needs conducting validity and reliability test to make sure that the tool is valid and reliable. Method: There were two types of validity test used, content validity test and construct validity test as well as reliability test, namely inter-rater reliability test and item covariance test. Participants were four clinical instructors and 116 nursing students. Result: Content validity test showed that two items must be added as part of assessment item, namely diciplines and faithness. Furthermore construct validity test showed that five items were not valid since they had pearson correlation score <0.3. However the items were included as consideration of nursing students’s PB. Inter-rater reliability test and item covariance reliability test showed that the tool was reliable with score 0.460 and 0.912 respectively. Discussion: The assessment tool can be applied to assess PB of nursing students since it valid and reliable. It needs to investigate the effectiveness of the tool in difference of PB of nursing students.Keywords: validity, reliability, assessment tool of PB (professional behaviour)

Download Full-text

Validation of the Equine Behaviour Assessment and Research Questionnaire (E-BARQ): A New Survey Instrument for Exploring and Monitoring the Domestic Equine Triad

Animals ◽

10.3390/ani10111982 ◽

2020 ◽

Vol 10 (11) ◽

pp. 1982 ◽

Cited By ~ 4

Author(s):

Kate Fenner ◽

Sarah Matlock ◽

Jane Williams ◽

Bethany Wilson ◽

Andrew McLean ◽

...

Keyword(s):

Construct Validity ◽

Quantitative Data ◽

Assessment Tool ◽

Questionnaire Validation ◽

Subjective Ratings ◽

Validity And Reliability ◽

Rater Reliability ◽

Validity Testing ◽

Significant Alignment ◽

Behavioural Assessment

The Equine Behaviour Assessment and Research Questionnaire (E-BARQ) was developed to obtain quantitative data on the domestic equine triad: training, management and behaviour. It can be taken repeatedly, thus collecting longitudinal data to enable evaluation of how changes in a horse’s training and management are reflected in that horse’s behaviour over time and how these changes can impact horse welfare in the longer term. Questionnaire validation and reliability were tested by determining (a) whether an owner’s subjective ratings of their horse’s problematic behaviours or undesirable temperament traits were reflected in the questionnaire scores obtained for that horse (construct validity), (b) whether two respondents, equally familiar with a particular horse, reported comparable scores for that horse through the questionnaire (inter-rater reliability), and (c) whether the same respondent, scoring the same horse after a known interval of time, recorded similar responses (intra-rater reliability). Construct validity testing of 1923 responses showed significant alignment between owners’ reported experience of focal horses’ behaviour and those horses’ E-BARQ scores, with scores varying from 1.13 to 1.34 for ridden horse behaviour (all p < 0.001) and from 1.06 to 1.43 for non-ridden horse behaviour (all p < 0.001). Inter-rater reliability testing of ten horse–rider pairs revealed that 203 of the 215 question items were significantly aligned (p < 0.001) when tested by two independent raters. Of the remaining 19 items, four had fair alignment (ĸ = 0.174–0.316; p = 0.281) and ten items, largely related to whether the horse shows behavioural signs related to anxiety when taken away from home, did not align (ĸ = 0; p = 1). Intra-rater reliability tests showed that the responses significantly aligned on all 215 question items tested (p < 0.001). The results of these tests confirmed the construct validity and reliability of E-BARQ as a standardised behavioural assessment tool for horses.

Download Full-text