scholarly journals Measuring the Master Adaptive Learner: Development and Internal Structure Validity Evidence for a New Instrument

2022 ◽  
Author(s):  
J. K. Stringer ◽  
Larry D. Gruppen ◽  
Michael S. Ryan ◽  
Samara B. Ginzburg ◽  
William B. Cutrer ◽  
...  
Keyword(s):  
Internal Structure ◽  
Validity Evidence ◽  
New Instrument

Factor Structure of the Scales for Assessing Emotional Disturbance : 3 Rating Scale for Students Identified With Emotional Disturbance

2021 ◽  
pp. 019874292110123
Author(s):  
Matthew C. Lambert ◽  
Douglas Cullinan ◽  
Michael H. Epstein ◽  
Jodie Martin
Keyword(s):  
Internal Structure ◽  
Emotional Disturbance ◽  
Factor Structure ◽  
Rating Scale ◽  
Factor Model ◽  
Distinct Group ◽  
Considerable Proportion ◽  
Evidence Based ◽  
Validity Evidence ◽  
Common Variance

This study examined the internal structure of the Scales for Assessing Emotional Disturbance-3 Rating Scale (RS), a teacher-completed RS developed to measure emotional disturbance (ED). As defined in U.S. law and regulations, ED involves five characteristics or patterns of behavioral and emotional maladaptation. RS data obtained on a sample of students with ED were used to examine validity evidence based on the internal structure of the assessment. Of particular interest was the extent to which multivariate factors derived from the RS data conform to the five characteristics of ED stated in the definition. Results indicate that the RS data fit a 5-factor model reasonably well. A subsequent bifactor analysis identified a considerable proportion of common variance across factors, suggesting the presence of a strong general ED factor, two distinct group factors (Inability to Learn and Inappropriate Behavior), and three weak group factors. The findings provided evidence of the validity of the SAED-3 RS based on internal structure and pointed to support for use of the RS in contributing to the process of determining whether a student qualifies for the ED education disability. Implications for improved research on the nature of ED and how students with ED can be better served are discussed.

scholarly journals LO84: Ready to run the show: development of a new instrument for assessing resident competence in the emergency department

CJEM ◽  
2019 ◽  
Vol 21 (S1) ◽  
pp. S38
Author(s):  
W. Cheung ◽  
W. Gofton ◽  
T. Wood ◽  
M. Duffy ◽  
S. Dewhirst ◽  
...  
Keyword(s):  
Emergency Department ◽  
Emergency Medicine ◽  
Nominal Group Technique ◽  
Nominal Group ◽  
Independent Practice ◽  
Validity Evidence ◽  
Training Level ◽  
Multiple Sources ◽  
Treatment Area ◽  
New Instrument

Innovation Concept: The outcome of emergency medicine training is to produce physicians who can competently run an emergency department (ED) shift. While many workplace-based ED assessments focus on discrete tasks of the discipline, others emphasize assessment of performance across the entire shift. However, the quality of assessments is generally poor and these tools often lack validity evidence. The use of entrustment scale anchors may help to address these psychometric issues. The aim of this study was to develop and gather validity evidence for a novel tool to assess a resident's ability to independently run an ED shift. Methods: Through a nominal group technique, local and national stakeholders identified dimensions of performance reflective of a competent ED physician. These dimensions were included in a new tool that was piloted in the Department of Emergency Medicine at the University of Ottawa during a 4-month period. Psychometric characteristics of the items were calculated, and a generalizability analysis used to determine the reliability of scores. An ANOVA was conducted to determine whether scores increased as a function of training level (junior = PGY1-2, intermediate = PGY3, senior = PGY4-5), and varied by ED treatment area. Safety for independent practice was analyzed with a dichotomous score. Curriculum, Tool or Material: The developed Ottawa Emergency Department Shift Observation Tool (O-EDShOT) includes 12-items rated on a 5-point entrustment scale with a global assessment item and 2 short-answer questions. Eight hundred and thirty-three assessment were completed by 78 physicians for 45 residents. Mean scores differed significantly by training level (p < .001) with junior residents receiving lower ratings (3.48 ± 0.69) than intermediate residents who received lower ratings (3.98 ± 0.48) than senior residents (4.54 ± 0.42). Scores did not vary by ED treatment area (p > .05). Residents judged to be safe to independently run the shift had significantly higher mean scores than those judged not to be safe (4.74 ± 0.31 vs 3.75 ± 0.66; p < .001). Fourteen observations per resident, the typical number recorded during a 1-month rotation, were required to achieve a reliability of 0.80. Conclusion: The O-EDShOT successfully discriminated between junior, intermediate and senior-level residents regardless of ED treatment area. Multiple sources of evidence support the O-EDShOT producing valid scores for assessing a resident's ability to independently run an ED shift.

scholarly journals Padjadjaran Interest Inventory: Evaluation of Psychometric Properties

2019 ◽  
Vol 46 (1) ◽  
pp. 19
Author(s):  
Whisnu Yudiana ◽  
Ilham Phalosa Reswara ◽  
Sudarmo Wiyono ◽  
R Urip Purwono
Keyword(s):  
Psychometric Properties ◽  
Career Exploration ◽  
Future Research ◽  
Validity Evidence ◽  
Good Reliability ◽  
Interest Inventory ◽  
New Instrument ◽  
Confirmatory Factor ◽  
Gender And It ◽  
Large Groups

This research aims to evaluate psychometric properties of a new instrument for measuring vocational interest: Padjadjaran Interest Inventory (PII). There were 2,648 participants in this study, consisting of workers, high school, and university students, with gender proportion of 1,014 (38.3%) males and 1,634 (61.7%) females. This research used descriptive statistic test, t test and MANOVA for gender differences, reliability coefficients and validity evidence by using confirmatory factor analysis (CFA). The results showed that PII has a good psychometric properties: it has good reliability and valid internal structure; it is standardized by gender; and it is applicable for large groups with relative ease. PII can be used for career exploration. Limitation of this study was discussed for future research.

scholarly journals Development of the Inventory of Supporting for Socio-Emotional Skills, evidence of internal structure controlling for acquiescence

2020 ◽  
Vol 37 ◽  
Author(s):  
Felipe Valentini ◽  
Leonardo de Barros Mose ◽  
Isabella de Sousa Ramos ◽  
Nathalia Martins da Conceição
Keyword(s):  
High School ◽  
High School Students ◽  
Internal Structure ◽  
Emotional Development ◽  
Skills Development ◽  
Validity Evidence ◽  
School Students ◽  
Emotional Skills ◽  
Final Structure ◽  
Random Intercept

Abstract This study aims to develop the Inventory of Supporting for Socio-Emotional Skills, which is an instrument for assessing the support for socio-emotional development provided by teachers, rated by students. It also aims to gather validity evidence of the internal structure of Inventory of Supporting for Socio-Emotional Skills and to control acquiescence. We elaborated the items, which content were evaluated by six experts. For the internal structure, a sample composed of elementary and high-school students (N = 443) was used to answer the instrument. The instrument presented good psychometric parameters after controlling for acquiescence using the Random Intercept Model. The final structure was set as unidimensional. We suggest controlling acquiescence to avoid biased scores. In addition, we make an inventory available that can be helpful for socio-emotional skills development.

Validity Evidence based on Internal Structure of Scores of the Emotional Quotient-Inventory: Youth Version Short (EQ-i: YV-S) in a Spanish Sample

2016 ◽  
Vol 19 ◽  
Author(s):  
Igor Esnaola ◽  
John Freeman ◽  
Marta Sarasa ◽  
Arantza Fernández-Zabala ◽  
Inge Axpe
Keyword(s):  
Factor Analysis ◽  
Stress Management ◽  
Internal Structure ◽  
Structural Equation ◽  
Reliability And Validity ◽  
Self Concept ◽  
Equation Modeling ◽  
Evidence Based ◽  
Validity Evidence ◽  
Power Prediction

AbstractThe purpose of this study was to analyze the reliability and validity evidence of scores on the Spanish version of EQ-i: YV-S in Spanish adolescents. The total sample was comprised of 508 participants from Grades 7 to 12, 241 males (47.4%) and 267 females (52.6%), each of whom completed the questionnaires on two separate occasions. Three [intrapersonal (α = .83, CR = .86, and McDonald Omega = .86), stress management (α = .83, CR = .86, and McDonald Omega = .85) and adaptability (α = .82, CR = .85, and McDonald Omega = .85)] of the four scales had acceptable internal consistency. Exploratory Factor Analysis (EFA) and Confirmatory Factor Analysis (CFA) were used with FACTOR and EQS version 6.1 software to examine validity evidence based on internal structure drawn from the scores on the EQ-i: YV-S, supporting the multidimensionality of the questionnaire. Three models were tested; the best fit to the data was the hierarchical model (S-Bχ2/ df = 2.11, CFI = .93 and RMSEA = .047), which hypothesized that the four specific factors (interpersonal, intrapersonal, stress management, and adaptability) were explained with a second-order factor, Emotional-Social-Intelligence (ESI). Finally, significant positive correlations were found between general self-concept and EQ-i: YV-S [interpersonal (r = .153, p < .001), intrapersonal (r = .235, p < .001), stress management (r = .145, p < .001), adaptability (r = .311, p < .001) and ESI (r = .360, p < .001)]; ESI showed significant direct power prediction of the general self-concept (.52) as demonstrated through structural equation modeling.

scholarly journals Development and validity evidence of the Two-Dimensional Sexual Sensation Seeking Scale

2020 ◽  
Vol 52 ◽  
Author(s):  
Rodrigo Ferrer-Urbina ◽  
◽  
Patricio Mena-Chamorro ◽  
Patricio Zambrana ◽  
Cristian Ramírez ◽  
...  
Keyword(s):  
Internal Structure ◽  
Latin American ◽  
Sexual Behaviors ◽  
Sensation Seeking ◽  
Two Dimensions ◽  
Validity Evidence ◽  
Two Dimensional ◽  
Time Space ◽  
Sexual Sensation Seeking ◽  
Sensation Seeking Scale

Sexually transmitted infections (STIs), mainly HIV/AIDS, are acquired through risky sexual behaviors that have been associated with sexual sensation seeking. The purpose of this work is development a new scale for the assessment of sexual sensations seeking, with evidence of validity based on internal structure and relationship to other measures, for use in young people and adults in a Latin American context. An instrumental study was performed, with time-space sampling of students from the two Chilean cities with the highest rates of HIV. Final scale has 9 items to evaluate two dimensions: (1) sexual emotions seeking; and (2) tendency to sexual boredom. The identified structure provides good levels of reliability and presents validity evidence, based on the internal structure of the test, using CFA and ESEM. Two-dimensional sexual sensation seeking scale evidence proper psychometric properties to evaluate the seeking for sexual sensations in equivalents samples.

scholarly journals Psychometric Assessment of the Post-Secondary Student Stressors Index (PSSI)

2019 ◽  
Author(s):  
Brooke Linden ◽  
Heather Stuart
Keyword(s):  
Mental Health ◽  
Internal Structure ◽  
Secondary Students ◽  
Poor Mental Health ◽  
Secondary Student ◽  
Retest Reliability ◽  
Post Secondary ◽  
New Instrument ◽  
Test Retest Reliability ◽  
Post Secondary Students

Abstract Background: Previous research has linked excessive stress among post-secondary students to poor academic performance and poor mental health. Despite attempts to ameliorate mental health challenges at post-secondary institutions, there exists a gap in the evaluation of the specific sources of stress for students within the post-secondary setting. Methods: The goal of this study was to develop a new instrument to better assess the sources of post-secondary student stress. Over the course of two years, the Post-Secondary Student Stressors Index (PSSI) was created in collaboration with post-secondary students as co-developers and subject matter experts. In this study, we used a combination of individual cognitive interviews (n = 11), an online consensus survey modeled after a traditional Delphi method (n = 65), and an online pre- (n = 535) and post-test (n = 350) survey to psychometrically evaluate the PSSI using samples of students from Ontario, Canada. We collected four types of evidence for validity, including: content evidence, response processes evidence, internal structure evidence, and relations to other variables. The test-retest reliability of the instrument was also evaluated. Results: The PSSI demonstrated strong psychometric properties. Content validation and response processes evidence was derived from active student involvement throughout the development and refinement of the tool. Exploratory factor analysis suggested that the structure of the PSSI reflects the internal structure of an index, rather than a scale, as expected. Test-retest reliability of the instrument was comparable to existing, established instruments. Finally, the PSSI demonstrated good relationships with like measures of stress, distress, and resilience, in the hypothesized directions. Conclusions: The PSSI is a 46-item inventory that will allow post-secondary institutions to pinpoint the most severe and frequently occurring stressors on their campus. This knowledge will facilitate appropriate targeting of priority areas, and help institutions to better align their mental health promotion and mental illness prevention programming with the needs of their campus.

scholarly journals The relationship between learner engagement and teaching effectiveness: a novel assessment of student engagement in continuing medical education

2020 ◽  
Vol 20 (1) ◽  
Author(s):  
Christopher R. Stephenson ◽  
Sara L. Bonnes ◽  
Adam P. Sawatsky ◽  
Lukas W. Richards ◽  
Cathy D. Schleck ◽  
...  
Keyword(s):  
Medical Education ◽  
Factor Analysis ◽  
Internal Structure ◽  
Internal Consistency ◽  
Teaching Effectiveness ◽  
Internal Consistency Reliability ◽  
Validity Evidence ◽  
Response Process ◽  
Learner Engagement ◽  
The Relationship

Abstract Background Continuing medical education (CME) often uses passive educational models including lectures. However, numerous studies have questioned the effectiveness of these less engaging educational strategies. Studies outside of CME suggest that engaged learning is associated with improved educational outcomes. However, measuring participants’ engagement can be challenging. We developed and determined the validity evidence for a novel instrument to assess learner engagement in CME. Methods We conducted a cross-sectional validation study at a large, didactic-style CME conference. Content validity evidence was established through review of literature and previously published engagement scales and conceptual frameworks on engagement, along with an iterative process involving experts in the field, to develop an eight-item Learner Engagement Instrument (LEI). Response process validity was established by vetting LEI items on item clarity and perceived meaning prior to implementation, as well as using a well-developed online platform with clear instructions. Internal structure validity evidence was based on factor analysis and calculating internal consistency reliability. Relations to other variables validity evidence was determined by examining associations between LEI and previously validated CME Teaching Effectiveness (CMETE) instrument scores. Following each presentation, all participants were invited to complete the LEI and the CMETE. Results 51 out of 206 participants completed the LEI and CMETE (response rate 25%) Correlations between the LEI and the CMETE overall scores were strong (r = 0.80). Internal consistency reliability for the LEI was excellent (Cronbach’s alpha = 0.96). To support validity to internal structure, a factor analysis was performed and revealed a two dimensional instrument consisting of internal and external engagement domains. The internal consistency reliabilities were 0.96 for the internal engagement domain and 0.95 for the external engagement domain. Conclusion Engagement, as measured by the LEI, is strongly related to teaching effectiveness. The LEI is supported by robust validity evidence including content, response process, internal structure, and relations to other variables. Given the relationship between learner engagement and teaching effectiveness, identifying more engaging and interactive methods for teaching in CME is recommended.

Sign in / Sign up

Export Citation Format

Share Document