scholarly journals Providing Validation Evidence for a Clinical-Science Module: Improving Testing Reliability with Quizzes

2021 ◽  
Vol 12 (1) ◽  
pp. 17
Author(s):  
Michael Peeters ◽  
M Kenneth Cor ◽  
Erik Maki
Keyword(s):  
Instructional Strategy ◽  
Generalizability Theory ◽  
Relative Weight ◽  
Clinical Science ◽  
Equal Weight ◽  
Multiple Choice Questions ◽  
High Stakes ◽  
Cardiovascular Therapeutics ◽  
Relative Weighting ◽  
Composite Reliability

Description of the Problem: High-stakes decision-making should have sound validation evidence; reliability is vital towards this. A short exam may not be very reliable on its own within didactic courses, and so supplementing it with quizzes might help. But how much? This study’s objective was to understand how much reliability (for the overall module-grades) could be gained by adding quiz data to traditional exam data in a clinical-science module. The Innovation: In didactic coursework, quizzes are a common instructional strategy. However, individual contexts/instructors can vary quiz use formatively and/or summatively. Second-year PharmD students took a clinical-science course, wherein a 5-week module focused on cardiovascular therapeutics. Generalizability Theory (G-Theory) combined seven quizzes leading to an exam into one module-level reliability, based on a model where students were crossed with items nested in eight fixed testing occasions (mGENOVA used). Furthermore, G-Theory decision-studies were planned to illustrate changes in module-grade reliability, where the number of quiz-items and relative-weighting of quizzes were altered. Critical Analysis: One-hundred students took seven quizzes and one exam. Individually, the exam had 32 multiple-choice questions (MCQ) (KR-20 reliability=0.67), while quizzes had a total of 50MCQ (5-9MCQ each) with most individual quiz KR-20s less than or equal to 0.54. After combining the quizzes and exam using G-Theory, estimated reliability of module-grades was 0.73; improved from the exam alone. Doubling the quiz-weight, from the syllabus’ 18% quizzes and 82% exam, increased the composite-reliability of module-grades to 0.77. Reliability of 0.80 was achieved with equal-weight for quizzes and exam. Next Steps: Expectedly, more items lent to higher reliability. However, using quizzes predominantly formatively had little impact on reliability, while using quizzes more summatively (i.e., increasing their relative-weight in module-grade) improved reliability further. Thus, depending on use, quizzes can add to a course’s rigor.

scholarly journals Validation Evidence from using Generalizability Theory in a Basic-Science Course

2021 ◽  
Vol 12 (1) ◽  
pp. 16
Author(s):  
Michael Peeters ◽  
M Kenneth Cor ◽  
Sai Boddu ◽  
Jerry Nesamony
Keyword(s):  
Decision Making ◽  
Generalizability Theory ◽  
First Year ◽  
High Stakes ◽  
Final Exam ◽  
Course Grades ◽  
Added Benefit ◽  
Acceptable Reliability ◽  
Composite Reliability ◽  
Better Than

Description of the Problem: Reliability is critical validation evidence on which to base high-stakes decision-making. Many times, one exam in a didactic course may not be acceptably reliable on its own. But how much might multiple exams add when combined together? The Innovation: To improve validation evidence towards high-stakes decision-making, Generalizability Theory (G-Theory) can combine reliabilities from multiple exams into one composite-reliability (G_String IV software). Further, G-Theory decision-studies can illustrate changes in course-grade reliability, depending on the number of exams and exam-items. Critical Analysis: 101 first-year PharmD students took two midterm-exams and one final-exam in a pharmaceutics course. Individually, Exam1 had 50MCQ (KR-20=0.69), Exam2 had 43MCQ (KR-20=0.65), and Exam3 had 67MCQ (KR-20=0.67). After combining exam occasions using G-Theory, the composite-reliability was 0.71 for overall course-grades—better than any exam alone. Remarkably, increased numbers of exam occasions showed fewer items per exam were needed, and fewer items over all exams, to obtain an acceptable composite-reliability. Acceptable reliability could be achieved with different combinations of number of MCQs on each exam and number of exam occasions. Implications: G-Theory provided reliability critical validation evidence towards high-stakes decision-making. Final course-grades appeared quite reliable after combining multiple course exams—though this reliability could and should be improved. Notably, more exam occasions allowed fewer items per exam and fewer items over all the exams. Thus, one added benefit of more exam occasions for educators is developing fewer items per exam and fewer items over all exams.

Best of Five MCQs for the Acute Medicine SCE

2016 ◽  
Keyword(s):  
Multiple Choice ◽  
Multiple Choice Questions ◽  
Self Assessment ◽  
High Quality ◽  
High Stakes ◽  
Acute Medicine ◽  
Acute Conditions

Best of Five MCQs for the Acute Medicine SCE is a new revision resource designed specifically for this high-stakes exam. Containing over 350 Best of Five multiple choice questions, this dedicated guide will help candidates to prepare successfully. The content mirrors the SCE in Acute Medicine Blueprint to ensure candidates are fully prepared for all the topics that may appear in the exam. Topics range from how to manage acute problems in cardiology or neurology to managing acute conditions such as poisoning. All answers have full explanations and further reading to ensure high quality self-assessment and quick recognition of areas that require further study.

scholarly journals The frequency of item writing flaws in multiple-choice questions used in high stakes nursing assessments

2006 ◽  
Vol 26 (8) ◽  
pp. 662-671 ◽  
Author(s):  
Marie Tarrant ◽  
Aimee Knierim ◽  
Sasha K. Hayes ◽  
James Ware
Keyword(s):  
Multiple Choice ◽  
Multiple Choice Questions ◽  
High Stakes ◽  
Item Writing

What does it mean to be civic-ready? Uncovering citizenship conceptualizations in US states that require the ‘citizenship test’

2020 ◽  
pp. 174619792094997
Author(s):  
Aaron Paul Johnson
Keyword(s):  
The United States ◽  
State Standards ◽  
American Identity ◽  
Test Model ◽  
Multiple Choice Questions ◽  
High Stakes ◽  
Civics Education ◽  
Exclusionary Practices ◽  
Us States ◽  
Attention To Diversity

There is a growing trend within civics education within the United States to adopt the USCIS naturalization civics exam (commonly referred to as the ‘citizenship test’), which consists of 100 multiple choice questions, to determine the civic readiness of students. This study explores civics education within eight US states that have adopted the ‘citizenship test’ model through the lens of citizenship conceptualizations evidenced within their high school civics state standards. Utilizing a directed content analysis, the researcher located more than 230 standards within three citizenship discourse categories: civic republican, liberal, and reconstructionist. Citizenship within the examined state standards, was largely conceptualized as civic republican in orientation (96%) and nested within untroubled assumptions of US society and desires for a common American identity that often crowded out attention to diversity and conflict and in doing so, mirrored exclusionary practices common to standardized, high-stakes assessments including the USCIS naturalization civics exam.

scholarly journals Policy in the Way of Practice: How Assessment Legislation Is Affecting Social Studies Curriculum and Instruction in Ohio

2011 ◽  
Vol 6 (7) ◽  
Author(s):  
Thomas Misco ◽  
Nancy Patterson ◽  
Frans Doppen
Keyword(s):  
Social Studies ◽  
Citizenship Education ◽  
Instructional Strategy ◽  
Curriculum And Instruction ◽  
High Stakes Testing ◽  
National Context ◽  
Social Studies Curriculum ◽  
High Stakes ◽  
The Social

In a national context of standards and high-stakes testing, concerns are emerging about challenges to the already tenuous position of the citizenship mission in the social studies curriculum. In this qualitative study, the authors administered a survey to social studies teachers in Ohio and conducted follow-up interviews focusing on the present purposes of social studies and the ways in which standards and testing are affecting instructional practice. The findings reveal a perception of standards as being of high quality, yet ultimately undermined through changes in scope and se-quence, narrowing of the curriculum, and a paucity of time to enact them. In addition, respondents indicated that high-stakes testing has become the primary curricular focus, which impacts instructional strategy decision making and frustrates citizenship education.

A multi-criteria decision making model for outsourcing maintenance services

Facilities ◽  
2015 ◽  
Vol 33 (3/4) ◽  
pp. 229-244 ◽  
Author(s):  
Mohammad A. Hassanain ◽  
Sadi Assaf ◽  
Abdul-Mohsen Al-Hammad ◽  
Ahmed Al-Nehmi
Keyword(s):  
Decision Making ◽  
Air Conditioning ◽  
Relative Weight ◽  
Equal Weight ◽  
Multi Criteria Decision Making ◽  
Content Type ◽  
Air Conditioning Systems ◽  
Decision Making Model ◽  
Outsourcing Decision

Purpose – The purpose of this paper is to present the development of a multi-criteria decision-making model for use by maintenance managers to consider before making a decision on outsourcing. Design/methodology/approach – Thirty-eight factors were identified for outsourcing maintenance services. These factors were grouped under six categories, namely: “strategic”, “management”, “technological”, “quality”, “economic” and “function characteristics”. The Analytic Hierarchy Process, as a multi-criteria decision-making model, was introduced and applied as an approach for maintenance managers in Saudi Arabian universities to consider before making a decision on outsourcing. A case study on the outsourcing decision of maintenance services of air-conditioning systems was carried out to apply the developed model. Findings – Data analysis indicated that all outsourcing decision groups of factors have almost equal weight, with the “quality” group of factors having the highest weight and the “technological” group of factors having the least weight. Further, the analysis indicated, in general, that the recommended decision for the maintenance managers is to outsource. However, an application of the developed model through a case study on the outsourcing of maintenance services of air-conditioning systems showed that the recommended action is not to outsource. Originality/value – The presented approach in this paper could be of practical benefit to maintenance managers in their decision making of whether or not to outsource maintenance services. The factors in the model were identified through a literature survey of research carried out in different countries. Therefore, the model could be applied in different settings, depending on the relative weight of the factors by the users.

scholarly journals Faculty Perceptions on Objective Structured Clinical Exam in Dental Education

2021 ◽  
pp. 445-456
Author(s):  
Othman Wali ◽  
Amit Vanka ◽  
Shanthi Vanka
Keyword(s):  
Cognitive Skills ◽  
Faculty Perceptions ◽  
Likert Scale ◽  
Multiple Choice Questions ◽  
High Stakes ◽  
Clinical Exam ◽  
Objective Structured Clinical Exam ◽  
Minimum Number ◽  
Competency Based ◽  
Multiple Domains

Structured Clinical Exam (OSCE) uses standardized content and procedures to assess students across multiple domains of learning. The study is aimed to assess knowledge, attitudes, practices and observations of dental faculty on OSCE. The survey was distributed into dental faculty members in randomly selected government and private institutions in Saudi Arabia. The questionnaire was pre-tested and consisted of 4 categories including general characteristics of respondents, knowledge on utility of OSCE in curriculum and it’s reliability, attitudes regarding OSCE on a 5 point Likert scale, practices and observations on OSCE on Multiple choice questions (both single answer and multiple answer) and responses on a 5 point Likert scale. The sample size was determined to be 93 and the survey was sent electronically to 10 institutes. 101 complete responses from 7 institutions were considered from the 122 received. Faculty participation in OSCE was high within evaluators 94% (n=94) and administrators 61% (n=61). Majority of respondents (62%) believed that OSCE is most suited for competency based education, to assess cognitive skills (73%) and diagnostic interpretation (79%). Reliability of OSCE can be increased by standardization of evaluators (77%) with highest number believing that 6-8 stations (42%) are the minimum required in an OSCE.  Institution guidelines (49%) coupled with workshops (47%) was the preferred method of preparation for OSCE. Majority felt that OSCE is most suitable for high stakes exams (mean=3.37) and it is an indispensable part of dental assessment (mean=3.78). Minimum number of stations for adequate reliability was reported to be lesser that in reported literature, specially so for high stakes assessments. Logistics required for arranging an  OSCE  and  difficulty in standardized patients, may suggest that OSCE should be used in select situations.

Sign in / Sign up

Export Citation Format

Share Document