Analisis Properti Psikometri Career Adapt-Ability Scale pada Siswa Sekolah Menengah Kejuruan (SMK)

Sebagai dampak dari perkembangan dunia kerja yang semakin fleksibel, individu dituntut untuk memiliki kemampuan adaptasi dalam konteks karier yang disebut dengan career adaptability. Career Adapt-Ability Scale merupakan alat ukur yang disusun oleh Savickas & Porfeli ditujukan untuk mengukur career adaptability. Penelitian ini bertujuan untuk mengevaluasi properti psikometri Career Adapt-Ability Scale yang telah diadaptasi ke Bahasa Indonesia. Subjek penelitian ini berjumlah 121 orang dengan rentang usia 14 – 17 tahun. Penelitian ini dibagi ke dalam dua tahapan: Pertama, melakukan uji reliabilitas melalui internal consistency dengan mengevaluasi nilai koefisien Cronbach’s alpha dan melalui analisis item discrimination dengan mengevaluasi nilai corrected-item total correlation pada masing-masing item. Hasilnya menunjukkan bahwa alat ukur CAAS dapat diandalkan dan memiliki reliabilitas yang baik (alfa = 0,746 – 0,923). Kedua, menguji validitas konstruk menggunakan confirmatory factor analysis, hasilnya menunjukkan bahwa data yang ada sesuai dengan model teori career adaptability dan setiap item pada alat ukur CAAS terbukti signifikan dalam mengukur dimensinya pada konstruk career adaptability. Dengan demikian, disimpulkan bahwa alat ukur CAAS ini dapat diandalkan untuk mengukur career adaptability

FMAQ-S: Development of a Short Form of the Family Medicine Attitudes Questionnaire

Introduction: Identifying and training students who choose family medicine careers is essential to meeting primary care workforce needs in the United States. Medical students’ positive attitudes toward family medicine are associated with students’ choice of family medicine as a specialty. This study sought to refine a previously tested questionnaire assessing US medical students’ attitudes toward family medicine by shortening the questionnaire to make it more useful in educational practice and research settings. Methods: We refined our existing 14-item questionnaire by item analysis and validation. We conducted item analysis using a graded response model approach after identifying the unidimensionality of the original scale. We selected items based on their item discrimination parameters and item information levels, and calculated the correlation between specialty choice and family medicine attitudes score to evaluate criterion validity. Results: Exploratory factor analyses indicated the questionnaire is unidimensional. Among the original 14 items, 10 items had high item discrimination parameters and low standard error of measurement. These 10 items contribute the most to distinguishing individuals’ differences in family medicine attitudes and were selected for inclusion in the short-form questionnaire (FMAQ-S). The point-biserial correlation between the short-form scale and students’ choice of family medicine was 0.378, which provides supporting evidence for criterion validity. Conclusion: The FMAQ-S is a concise and validated measure for assessing medical student attitudes toward family medicine. This abbreviated questionnaire can be used by medical educators to identify students for specific programming or interventions intended to support family medicine specialty choice.

Item Response Theory for Psychometric Properties of the SNOT-22 (22-Item Sinonasal Outcome Test)

Objective The SNOT-22 (22-item Sinonasal Outcome Test) is a high-quality outcome measure that assesses chronic rhinosinusitis–specific quality of life. The aim of this study was to gain greater insight into the information provided by the SNOT-22 by determining its item-based psychometric properties. Study Design Retrospective cohort study. Setting Tertiary care academic centers. Methods This study used a previously described data set of the SNOT-22 completed by 800 patients with chronic rhinosinusitis. Item response theory graded response models were used to determine parameters reflecting item discrimination, difficulty, and information provided by each item toward the SNOT-22 subdomain to which it belonged. Results The unconstrained graded response model fitted the SNOT-22 data best. Item discrimination parameters and total information provided showed the greatest variability within the nasal subdomain, and the item related to sense of smell/taste demonstrated the lowest discrimination and provided the least amount of information overall. The dizziness item provided disparately lower total information and discrimination in the otologic/facial pain subdomain. Items in the sleep and emotional subdomains generally provided high discrimination. While items in the nasal, sleep, and otologic/facial pain subdomains spanned all levels of difficulty, emotional subdomain items covered higher levels of difficulty, indicating greater information provided at higher levels of disease severity. Conclusion The item-specific psychometric properties of the SNOT-22 support it as a high-quality instrument. Our results suggest the need and possibility for revision of the smell/taste dysfunction item, for example its wording, to improve its ability to discriminate among the different levels of disease burden.

ITEM ANALYSIS OF AN ENGLISH SUMMATIVE TEST

While almost half of the teachers’ activities are assessing their students, they are not well-prepared with assessment literacy training. Hence, they are unable to produce good tests to measure students’ level of knowledge and skills. This study is aimed at analyzing item difficulty and item discrimination of a test made by an English teacher at a junior high school in Kupang. It was descriptive qualitative research and the instruments of the research were test items, answer keys, and students’ answer sheets. For the difficulty index, it was revealed that more than half of the test items were easy, while only 2% of the test items were difficult. In terms of the discrimination index, it was found that only 10% of the test items were excellent and most of the test items (46%) were poor. These findings indicated that the English test had a poor item difficulty index and low item discrimination index. Hence, it did not fulfill the criteria of a good test and could not measure students’ true ability. It is highly recommended for the teachers to improve the test items and for the government to provide assessment training for the teachers so that they can produce good tests.

Consequences of Ignoring Guessing Effects on Measurement Invariance Analysis

Pseudo-guessing parameters are present in item response theory applications for many educational assessments. When sample size is not sufficiently large, the guessing parameters may be ignored from the analysis. This study examines the impact of ignoring pseudo-guessing parameters on measurement invariance analysis, specifically, on item difficulty, item discrimination, and mean and variance of ability distribution. Results show that when non-zero guessing parameters are ignored from the measurement invariance analysis, item discrimination estimates tend to decrease particularly for more difficult items, and item difficulty estimates decrease unless the items are highly discriminating and difficult. As the guessing parameter increases, the size of the decrease in item discrimination and difficulty tends to increase, and the estimated mean and variance of ability distribution tend to be inaccurate. When two groups have heterogeneous ability distributions, ignoring the guessing parameter affects the reference group and the focal group differently. Implications of result findings are discussed.

Assessment Question Characteristics Predict Medical Student Performance in General Pathology

Context.— Evaluation of medical curricula includes appraisal of student assessments in order to encourage deeper learning approaches. General pathology is our institution's 4-week, first-year course covering universal disease concepts (inflammation, neoplasia, etc). Objective.— To compare types of assessment questions and determine which characteristics may predict student scores, degree of difficulty, and item discrimination. Design.— Item-level analysis was employed to categorize questions along the following variables: type (multiple choice question or matching answer), presence of clinical vignette (if so, whether simple or complex), presence of specimen image, information depth (simple recall or interpretation), knowledge density (first or second order), Bloom taxonomy level (1–3), and, for the final, subject familiarity (repeated concept and, if so, whether verbatim). Results.— Assessments comprised 3 quizzes and 1 final exam (total 125 questions), scored during a 3-year period (total 417 students) for a total 52 125 graded attempts. Overall, 44 890 attempts (86.1%) were correct. In multivariate analysis, question type emerged as the most significant predictor of student performance, degree of difficulty, and item discrimination, with multiple choice questions being significantly associated with lower mean scores (P = .004) and higher degree of difficulty (P = .02), but also, paradoxically, poorer discrimination (P = .002). The presence of a specimen image was significantly associated with better discrimination (P = .04), and questions requiring data interpretation (versus simple recall) were significantly associated with lower mean scores (P = .003) and a higher degree of difficulty (P = .046). Conclusions.— Assessments in medical education should comprise combinations of questions with various characteristics in order to encourage better student performance, but also obtain optimal degrees of difficulty and levels of item discrimination.

Outcomes of Remote Pathology Instruction in Student Performance and Course Evaluation

The coronavirus disease 2019 (COVID-19) pandemic has disrupted undergraduate medical education, including preclinical class-based courses, by requiring social distancing and essentially eliminating in-person teaching. The aim of this study was to compare student performance and satisfaction before and after implementation of remote instruction in a first-year introductory pathology course. Assessments (3 quizzes, 1 practical exam, and 1 final) were compared between courses given before (January 2020) and during (January 2021) the COVID-19 pandemic, in terms of mean scores, degree of difficulty, and item discrimination, both overall and across different question types. Students’ evaluations of the course (Likert scale-based) were also compared between the 2 years. Significantly higher mean scores were observed during remote instruction (compared to the prior, in-person year) on verbatim-repeated questions (94.9 ± 8.8 vs 89.4 ± 12.2; P = .002) and on questions incorporating a gross specimen image (88.4 ± 7.5 vs 84.4 ± 10.3; P = .007). The percentage of questions that were determined to be moderate/hard in degree of difficulty and good/very good in item discrimination remained similar between the 2 time periods. In the practical examination, students performed significantly better during remote instruction on questions without specimen images (96.5 ± 7.0 vs 91.2 ± 15.2; P = .004). Finally, course evaluation metrics improved, with students giving a higher mean rating value in each measured end point of course quality during the year of remote instruction. In conclusion, student performance and course satisfaction generally improved with remote instruction, suggesting that the changes implemented, and their consequences, should perhaps inform future curriculum improvements.

Item Analysis of English Final Semester Test

Numerous studies have been conducted on the item test analysis in English test. However, investigation on the characteristics of a good test of English final semester test is still rare in several districts in East Java. This research sought to examine the quality of the English final semester test in the academic year of 2018/2019 in Ponorogo. A total of 151 samples in the form of students’ answers to the test were analysed based on item difficulty, item discrimination, and distractors’ effectiveness using Quest program. This descriptive quantitative research revealed that the test does not have good proportion among easy, medium, and difficult item. In the item discrimination, the test had 39 excellent items (97.5%) which meant that the test could discriminate among high and low achievers. Besides, the distractors could distract students since there were 32 items (80%) that had effective distractors. The findings of this research provided insights that item analysis became important process in constructing test. It related to find the quality of the test that directly affects the accuracy of students’ score.

The quality of an English summative test of a public junior high school, Kupang-NTT

This study aimed at examining the quality of an English summative test of grade VII in a public school located in Kupang. Particularly, this study examined content validity, reliability, and conducted item analysis including item validity, item difficulty, item discrimination, and distracter effectiveness. This study was descriptive evaluative research with documentation to collect data. The data was analyzed quantitatively except for content validity, which was done qualitatively. Content validity was analyzed by matching the test items with materials stated in the curriculum. The findings revealed that the English summative test had a high content validity. The reliability was estimated by applying the Kuder-Richardson’s formula (K-R20). The result showed that the test was reliable and very good for a classroom test. The item analysis was conducted by using ITEMAN 3.0. and it revealed that the the test was mostly constructed by easy items, most of the items could discriminate the students, most distracters were able to perform well, and the most of items were valid.