Toward an Evidence-Based Approach to Building Evaluation Capacity

2021 ◽  
Vol 36 (1) ◽  
Author(s):  
Btissam El Hassar ◽  
Cheryl Poth ◽  
Rebecca Gokiert ◽  
Okan Bulut
Keyword(s):  
Structural Equation ◽  
Assessment Tool ◽  
Evidence Based ◽  
Validity Evidence ◽  
Factor Analyses ◽  
Future Directions ◽  
Evaluation Capacity ◽  
Organizational Evaluation ◽  
Equation Modelling ◽  
Valid Evidence

Organizations are required to evaluate their programs for both learning and accountability purposes, which has increased the need to build their internal evaluation capacity. A remaining challenge is access to tools that lead to valid evidence supporting internal capacity development. The authors share practical insights from the development and use of the Evaluation Capacity Needs Assessment tool and framework and implications for using its data to make concrete decisions within Canadian contexts. The article refers to validity evidence generated from factor analyses and structural equation modelling and describes how applying the framework can be used to identify individual and organizational evaluation capac­ity strengths and gaps, concluding with practice considerations and future directions for this work.  

Validation evidence of the paediatric Objective Structured Assessment of Debriefing (OSAD) tool

2016 ◽  
Vol 2 (3) ◽  
pp. 61-67 ◽  
Author(s):  
Jane Runnacles ◽  
Libby Thomas ◽  
James Korndorffer ◽  
Sonal Arora ◽  
Nick Sevdalis
Keyword(s):  
Assessment Tool ◽  
Learning Experience ◽  
Pearson Correlation ◽  
Phase 1 ◽  
Evidence Based ◽  
Validity Evidence ◽  
Multiple Sources ◽  
Simulation Based ◽  
Expert Input

IntroductionDebriefing is essential to maximise the simulation-based learning experience, but until recently, there was little guidance on an effective paediatric debriefing. A debriefing assessment tool, Objective Structured Assessment of Debriefing (OSAD), has been developed to measure the quality of feedback in paediatric simulation debriefings. This study gathers and evaluates the validity evidence of OSAD with reference to the contemporary hypothesis-driven approach to validity.MethodsExpert input on the paediatric OSAD tool from 10 paediatric simulation facilitators provided validity evidence based on content and feasibility (phase 1). Evidence for internal structure validity was sought by examining reliability of scores from video ratings of 35 postsimulation debriefings; and evidence for validity based on relationship to other variables was sought by comparing results with trainee ratings of the same debriefings (phase 2).ResultsSimulation experts’ scores were significantly positive regarding the content of OSAD and its instructions. OSAD's feasibility was demonstrated with positive comments regarding clarity and application. Inter-rater reliability was demonstrated with intraclass correlations above 0.45 for 6 of the 7 dimensions of OSAD. The internal consistency of OSAD (Cronbach α) was 0.78. Pearson correlation of trainee total score with OSAD total score was 0.82 (p<0.001) demonstrating validity evidence based on relationships to other variables.ConclusionThe paediatric OSAD tool provides a structured approach to debriefing, which is evidence-based, has multiple sources of validity evidence and is relevant to end-users. OSAD may be used to improve the quality of debriefing after paediatric simulations.

Validity Evidence based on Internal Structure of Scores of the Emotional Quotient-Inventory: Youth Version Short (EQ-i: YV-S) in a Spanish Sample

2016 ◽  
Vol 19 ◽  
Author(s):  
Igor Esnaola ◽  
John Freeman ◽  
Marta Sarasa ◽  
Arantza Fernández-Zabala ◽  
Inge Axpe
Keyword(s):  
Factor Analysis ◽  
Stress Management ◽  
Internal Structure ◽  
Structural Equation ◽  
Reliability And Validity ◽  
Self Concept ◽  
Equation Modeling ◽  
Evidence Based ◽  
Validity Evidence ◽  
Power Prediction

AbstractThe purpose of this study was to analyze the reliability and validity evidence of scores on the Spanish version of EQ-i: YV-S in Spanish adolescents. The total sample was comprised of 508 participants from Grades 7 to 12, 241 males (47.4%) and 267 females (52.6%), each of whom completed the questionnaires on two separate occasions. Three [intrapersonal (α = .83, CR = .86, and McDonald Omega = .86), stress management (α = .83, CR = .86, and McDonald Omega = .85) and adaptability (α = .82, CR = .85, and McDonald Omega = .85)] of the four scales had acceptable internal consistency. Exploratory Factor Analysis (EFA) and Confirmatory Factor Analysis (CFA) were used with FACTOR and EQS version 6.1 software to examine validity evidence based on internal structure drawn from the scores on the EQ-i: YV-S, supporting the multidimensionality of the questionnaire. Three models were tested; the best fit to the data was the hierarchical model (S-Bχ2/ df = 2.11, CFI = .93 and RMSEA = .047), which hypothesized that the four specific factors (interpersonal, intrapersonal, stress management, and adaptability) were explained with a second-order factor, Emotional-Social-Intelligence (ESI). Finally, significant positive correlations were found between general self-concept and EQ-i: YV-S [interpersonal (r = .153, p < .001), intrapersonal (r = .235, p < .001), stress management (r = .145, p < .001), adaptability (r = .311, p < .001) and ESI (r = .360, p < .001)]; ESI showed significant direct power prediction of the general self-concept (.52) as demonstrated through structural equation modeling.

scholarly journals Validity Evidence for the Emergency Medicine Standardized Letter of Evaluation

2021 ◽  
Vol 13 (4) ◽  
pp. 490-499
Author(s):  
Paul Kukulski ◽  
James Ahn
Keyword(s):  
Emergency Medicine ◽  
Scoping Review ◽  
Assessment Tool ◽  
Successful Implementation ◽  
Validity Evidence ◽  
Response Process ◽  
Future Directions ◽  
Published Evidence ◽  
Structure Relation ◽  
Application Component

ABSTRACT Background The standardized letter of evaluation (SLOE) is the application component that program directors value most when evaluating candidates to interview and rank for emergency medicine (EM) residency. Given its successful implementation, other specialties, including otolaryngology, dermatology, and orthopedics, have adopted similar SLOEs of their own, and more specialties are considering creating one. Unfortunately, for such a significant assessment tool, no study to date has comprehensively examined the validity evidence for the EM SLOE. Objective We summarized the published evidence for validity for the EM SLOE using Messick's framework for validity evidence. Methods A scoping review of the validity evidence of the EM SLOE was performed in 2020. A scoping review was chosen to identify gaps and future directions, and because the heterogeneity of the literature makes a systematic review difficult. Included articles were assigned to an aspect of Messick's framework and determined to provide evidence for or against validity. Results There have been 22 articles published relating to validity evidence for the EM SLOE. There is evidence for content validity; however, there is a lack of evidence for internal structure, relation to other variables, and consequences. Additionally, the literature regarding response process demonstrates evidence against validity. Conclusions Overall, there is little published evidence in support of validity for the EM SLOE. Stakeholders need to consider changing the ranking system, improving standardization of clerkships, and further studying relation to other variables to improve validity. This will be important across GME as more specialties adopt a standardized letter.

Ageing-related influences on personal need for structure

2001 ◽  
Vol 25 (6) ◽  
pp. 482-490 ◽  
Author(s):  
Thomas M. Hess
Keyword(s):  
Structural Equation ◽  
Emotional Health ◽  
Social Activity ◽  
Factor Analyses ◽  
Confirmatory Factor Analyses ◽  
Adult Age ◽  
Personal Need For Structure ◽  
Equation Modelling ◽  
Confirmatory Factor ◽  
The Impact

The need for structure construct was examined in relation to adult age using the Personal Need for Structure scale (PNS; M.M. Thompson, M.E. Naccarato, & K. Parker, 1989). The results of a series of confirmatory factor analyses indicated that the two intercorrelated-factor structure of the PNS scale held up well across individuals aged 21 to 85, validating its use for the examination of ageing effects. Structural equation modelling analyses found that ageing was associated with lower levels of physical health and cognitive skill, which in turn were related to higher PNS scores. It was also found, however, that the impact of reductions in these resources on need for structure were counteracted by high levels of social activity and emotional health. The argument is made that ageing-related changes in personal resources impact everyday behaviour through changes in motivation, such as need for structure.

Self-Leadership and Physical Vitality

2010 ◽  
Vol 107 (2) ◽  
pp. 383-392 ◽  
Author(s):  
Günter F. Müller ◽  
Sibylle Georgianna ◽  
Gregor Roux
Keyword(s):  
Structural Equation Modelling ◽  
Structural Equation ◽  
Well Being ◽  
Leadership Strategies ◽  
Factor Analyses ◽  
Confirmatory Factor Analyses ◽  
Equation Modelling ◽  
Confirmatory Factor ◽  
Natural Rewards ◽  
Effective Behavior

The current study examined the extension of an approach to self-leadership using strategies which focus on (1) constructive thoughts, (2) natural rewards, and (3) effective behavior. The suggested extension refers to strategies which improve physical vitality, fitness, and well-being. A sample of 163 students answered scale items selected and composed to measure the four categories of self-leadership strategies. Confirmatory factor analyses and structural equation modelling indicated that strategies focusing on physical vitality, fitness, and well-being represent an independent category of self-leadership strategies. Implications for research and application were discussed.

scholarly journals Time spent shopping and consumer clothing purchasing behaviour

2018 ◽  
Vol 69 (2) ◽  
pp. 89-105 ◽  
Author(s):  
Mirela Mihić ◽  
Ivan-Damir Anić ◽  
Ivana Kursan Milaković
Keyword(s):  
Structural Equation ◽  
Single Point ◽  
Factor Analyses ◽  
Fashion Products ◽  
Equation Modelling ◽  
Purchasing Behaviour ◽  
Fashion Consciousness ◽  
Visual Merchandising ◽  
The Impact ◽  
Positive Effect

The purpose of this study is to explore the impact of various antecedents and behavioural consequences of consumers’ perceptions of time spent on shopping for clothing products. Examined antecedent variables are gender, age, income, fashion consciousness and visual merchandising, while outcomes include the number of clothing items purchased. The data, collected by consumer survey in Croatia, was analysed using exploratory and confi rmatory factor analyses, and structural equation modelling (SEM). The major fi ndings indicate that visual merchandising has the greatest and positive effect on time spent shopping. As expected, women and younger consumers spend more time shopping than men and older consumers. As for the outcomes, shopping time is positively related to consumer purchases. The study was conducted at a single point in time and was carried out while there was still crisis in Croatia, in a situation where consumers spend less on fashion products. The present study provides the insights into the determinantsand consequences of time expenditure and might help fashion retailers design more successfully retailing strategies to encourage consumers to spend more time on shopping in fashion stores, which in turn leads to higher sales.

scholarly journals Some goals just feel easier: Self-concordance leads to goal progress through subjective ease, not effort

2017 ◽  
Author(s):  
Kaitlyn M. Werner ◽  
Marina Milyavskaya ◽  
Emily Foxen-Craft ◽  
Richard Koestner
Keyword(s):  
Structural Equation Modelling ◽  
Structural Equation ◽  
Goal Pursuit ◽  
Future Directions ◽  
Goal Progress ◽  
Equation Modelling ◽  
Multilevel Structural Equation

The objective of the present study was to examine whether subjective ease of goal pursuit would mediate the relation between an individual’s motivation for pursuing a goal and their subsequent goal progress. Toward the beginning of a university semester, participants (n=176) identified three goals they planned to pursue throughout the semester and reported their motivation for pursuing each of them. Participants then indicated, at two monthly follow-ups, how easy and natural it felt to pursue these goals and how much effort they were putting into attaining them. At the end of the semester, participants reported on their goal progress. Within-person analyses indicated that self-concordant goals were perceived as being easier to pursue relative to an individual’s other goals. Using multilevel structural equation modelling, results indicated that subjective ease, but not effort, mediated the relation between motivation and goal progress, such that people were more likely to successfully accomplish self-concordant goals because pursuing those goals was perceived as being more effortless, and not because more effort was exerted. Discussion focuses on the implications and future directions for research on subjective effort and goal pursuit.

scholarly journals TURNING DATA INTO VALUABLE INSIGHTS: THE CASE STUDY IN AVIATION SECTOR COMPANY

2017 ◽  
Vol 5 ◽  
pp. 294-299
Author(s):  
Daiga Ergle ◽  
Iveta Ludviga ◽  
Agita Kalviņa
Keyword(s):  
Organizational Performance ◽  
Employee Engagement ◽  
Structural Equation ◽  
New Technologies ◽  
Evidence Based ◽  
Knowledge Based ◽  
Equation Modelling ◽  
Hr Function ◽  
Staff Engagement

Since the early 2000s, there is increasing pressure on Human Resource Departments to show their impact on organizational performance. This pressure is related to the shift from industrial based economies to knowledge based economies and positioning people as potential sources of competitive advantage, and to the rise of Evidence-Based Management (EBM), which requires making decisions based on data and analysis. New technologies have enabled HR departments to start a transition from HR metrics to HR Analytics, thus transforming from the traditional administrative HR function to a more strategic HR function that can express qualitative matters and its impact on organizational performance with numbers. This case study describes the implementation of HR analytics in an aviation sector company. Quantitative data gathered from an annual staff engagement survey are analyzed using a Structural Equation Modelling technique with Smart PLS software. The results show that the analysis offers insights which are much more valuable than traditional diagnosis of the level of employee engagement. Thus, management can trace an employee’s journey within the organization and be able to predict their behavior in relationship to the time spent in the organization. Moreover, the changing needs of employees are seen form the analysis and Evidence-Based Management can be implemented.

Meeting the Best Practice for Hearing Aid Verification in Children: Challenges and Future Directions

2019 ◽  
Vol 28 (4) ◽  
pp. 877-894
Author(s):  
Nur Azyani Amri ◽  
Tian Kar Quar ◽  
Foong Yen Chong
Keyword(s):  
Clinical Practice ◽  
Hearing Aids ◽  
Best Practice ◽  
Hearing Aid ◽  
Evidence Based ◽  
Future Directions ◽  
Major Barrier ◽  
Klang Valley ◽  
Age Range ◽  
Fitting Hearing Aids

Purpose This study examined the current pediatric amplification practice with an emphasis on hearing aid verification using probe microphone measurement (PMM), among audiologists in Klang Valley, Malaysia. Frequency of practice, access to PMM system, practiced protocols, barriers, and perception toward the benefits of PMM were identified through a survey. Method A questionnaire was distributed to and filled in by the audiologists who provided pediatric amplification service in Klang Valley, Malaysia. One hundred eight ( N = 108) audiologists, composed of 90.3% women and 9.7% men (age range: 23–48 years), participated in the survey. Results PMM was not a clinical routine practiced by a majority of the audiologists, despite its recognition as the best clinical practice that should be incorporated into protocols for fitting hearing aids in children. Variations in practice existed warranting further steps to improve the current practice for children with hearing impairment. The lack of access to PMM equipment was 1 major barrier for the audiologists to practice real-ear verification. Practitioners' characteristics such as time constraints, low confidence, and knowledge levels were also identified as barriers that impede the uptake of the evidence-based practice. Conclusions The implementation of PMM in clinical practice remains a challenge to the audiology profession. A knowledge-transfer approach that takes into consideration the barriers and involves effective collaboration or engagement between the knowledge providers and potential stakeholders is required to promote the clinical application of evidence-based best practice.

Sign in / Sign up

Export Citation Format

Share Document