scholarly journals Using Generalizability Theory and Many-Facet Rasch Model to Evaluate In-Basket Tests for Managerial Positions

2021 ◽  
Vol 12 ◽  
Author(s):  
Guangming Li ◽  
Yuxi Pan ◽  
Weijun Wang
Keyword(s):  
Rasch Model ◽  
Rating Scale ◽  
Generalizability Theory ◽  
Classical Test Theory ◽  
Interview Data ◽  
Test Theory ◽  
Personnel Recruitment ◽  
Communication And Coordination ◽  
Chinese Company ◽  
Managerial Positions

Purpose: This study aimed to analyze interview data collected from a series of in-basket tests during managerial personnel recruitment in a local Chinese company, taking advantage of the use of combination of Generalizability theory (GT) and Many-facet Rasch Model (MFRM), rather than the Classical Test Theory (CTT).Design/Methodology/Approach: Participants included 78 candidates (Mage = 37.76, SD = 6.36; 35.9% female) interviewed for eight managerial positions in a local Chinese company. Data were collected based on a series of 10 in-basket interview tests, and a 20-item rating scale (the in-basket test rating scale; IBTRS) was developed and piloted, and five expert raters rated the participants on their performance in five aspects (planning, communication and coordination, capital operations and management, analysis and problem solving, and empowerment and controlling). The data were analyzed using a crossed design of p × i × r, where p represents person, i represents item, and r represents rater. Effects of candidate (person), test item, rater, and the interaction of item and rater were examined.Findings: The use of the combination of GT and MFRM was able to provide accurate, comprehensive information over the process of in-basket interview tests. Specifically, GT analysis showed good generalization coefficient and reliability index (0.893 and 0.871, respectively), and the variation of candidates explained most of the total variance (53.22%). The candidates scored high in the dimension of empowerment and controlling. There were differences in the severity of raters. Three raters should be sufficient to ensure good scoring stability.Originality/Value: This study used the combination of GT and MFRM to assess the interview data instead of using a CTT approach.

scholarly journals Validation of the Spanish adaptation of the School Atitude Assessment Survey-Revised using multidimensional Rasch analysis

2016 ◽  
Vol 33 (1) ◽  
pp. 74 ◽  
Author(s):  
Alejandro Veas ◽  
Juan Luis Castejón ◽  
Raquel Gilar ◽  
Pablo Miñano
Keyword(s):  
Rasch Model ◽  
Rasch Analysis ◽  
Rating Scale ◽  
Item Difficulty ◽  
Classical Test Theory ◽  
Latent Trait ◽  
Principal Component ◽  
Analysis Data ◽  
Test Theory ◽  
Assessment Survey

<p>The School Attitude Assessment Survey-Revised (SAAS-R) was developed by McCoach &amp; Siegle (2003b) and validated in Spain by Author (2014) using Classical Test Theory. The objective of the current research is to validate SAAS-R using multidimensional Rasch analysis. Data were collected from 1398 students attending different high schools. Principal Component Analysis supported the multidimensional SAAS-R. The item difficulty and person ability were calibrated along the same latent trait scale. 10 items were removed from the scale due to misfit with the Rasch model. Differential Item Functioning revealed no significant differences across gender for the remaining 25 items. The 7-category rating scale structure did not function well, and the subscale goal valuation obtained low reliability values. The multidimensional Rasch model supported 25 item-scale SAAS-R measures from five latent factors. Therefore, the advantages of multidimensional Rasch analysis are demonstrated in this study.</p>

Revision of a Criterion-Referenced Vocabulary Test Using Generalizability Theory

2009 ◽  
Vol 31 (1) ◽  
pp. 81
Author(s):  
Takeaki Kumazawa
Keyword(s):  
Generalizability Theory ◽  
Classical Test Theory ◽  
Performance Testing ◽  
Error Variance ◽  
Test Theory ◽  
Vocabulary Test ◽  
Multiple Sources ◽  
Generalizability Coefficient ◽  
Criterion Referenced ◽  
Generalizability Study

Classical test theory (CTT) has been widely used to estimate the reliability of measurements. Generalizability theory (G theory), an extension of CTT, is a powerful statistical procedure, particularly useful for performance testing, because it enables estimating the percentages of persons variance and multiple sources of error variance. This study focuses on a generalizability study (G study) conducted to investigate such variance components for a paper-pencil multiple-choice vocabulary test used as a diagnostic pretest. Further, a decision study (D study) was conducted to compute the generalizability coefficient (G coefficient) for absolute decisions. The results of the G and D studies indicated that 46% of the total variance was due to the items effect; further, the G coefficient for absolute decisions was low. 古典的テスト理論は尺度の信頼性を測定するため広く用いられている。古典的テスト理論の応用である一般化可能性理論（G理論）は特にパフォーマンステストにおいて有効な分析手法であり、受験者と誤差の要因となる分散成分の割合を測定することができる。本研究では診断テストとして用いられた多岐選択式語彙テストの分散成分を測定するため一般化可能性研究（G研究）を行った。さらに、決定研究（D研究）では絶対評価に用いる一般化可能性係数を算出した。G研究とD研究の結果、項目の分散成分が全体の分散の46%を占め、また信頼度指数は高くなかった。

scholarly journals An item analysis according to the Rasch model of the German 12-item WHO Disability Assessment Schedule (WHODAS 2.0)

2021 ◽  
Author(s):  
Lusine Vaganian ◽  
Sonja Bussmann ◽  
Maren Boecker ◽  
Michael Kusch ◽  
Hildegard Labouvie ◽  
...  
Keyword(s):  
Cancer Patients ◽  
Rasch Model ◽  
Item Analysis ◽  
Classical Test Theory ◽  
Health Condition ◽  
Test Theory ◽  
World Health ◽  
Good Reliability ◽  
Whodas 2.0 ◽  
The Rasch Model

Abstract Purpose The World Health Organization Disability Assessent Schedule 2.0 (WHODAS 2.0) assesses disability in individuals irrespective of their health condition. Previous studies validated the usefulness of the WHODAS 2.0 using classical test theory. This study is the first investigating the psychometric properties of the 12-items WHODAS 2.0 in patients with cancer using item analysis according to the Rasch model. Methods In total, 350 cancer patients participated in the study. Rasch analysis of the 12-items version of the WHODAS 2.0 was conducted and included testing unidimensionality, local independence, and testing for differential item functioning (DIF) with regard to age, gender, type of cancer, presence of metastases, psycho-oncological support, and duration of disease. Results After accounting for local dependence, which was mainly found across items of the same WHODAS domain, satisfactory overall fit to the Rasch model was established (χ2 = 36.14, p = 0.07) with good reliability (PSI = 0.82) and unidimensionality of the scale. DIF was found for gender (testlet ‘Life activities’) and age (testlet ‘Getting around/Self-care’), but the size of DIF was not substantial. Conclusion Overall, the analysis results according to the Rasch model support the use of the WHODAS 2.0 12-item version as a measure of disability in cancer patients.

Psychometric evaluation of the Relative Mastery Scale: An Occupational Adaptation instrument

2021 ◽  
pp. 153944922110608
Author(s):  
Lorrie George-Paschal ◽  
Nancy E. Krusen ◽  
Chia-Wei Fan
Keyword(s):  
Rating Scale ◽  
Classical Test Theory ◽  
Correlation Coefficients ◽  
Measurement Model ◽  
Community Dwelling ◽  
Test Theory ◽  
Partial Credit Model ◽  
Validity And Reliability ◽  
Occupational Adaptation ◽  
Scale Properties

This study evaluated the psychometric properties of the Relative Mastery Scale (RMS). Valid and reliable client-centered instruments support practice in value-based health care and community-based settings. Participants were 368 community-dwelling adults aged 18 to 95 years. Researchers conducted validity and reliability examinations of the RMS using classical test theory and Rasch measurement model. A partial credit model allowed exploration of individual scale properties. Spearman’s correlation coefficients between items were statistically significant at the .01 level. Cronbach’s alpha coefficient was .94 showing strong internal consistency. In exploratory factor analysis, Factor 1 accounted for 71% of variance with an eigenvalue of 4.26. In Rasch analysis, the 5-point rating scale demonstrated adequate functioning, confirmed unidimensionality, and person/item separation. The RMS instrument demonstrates sound psychometric characteristics. A valid and reliable measure of internal occupational adaptation supports application to monitor progress of internal occupational adaptation across a variety of individuals.

scholarly journals Applicability of Item Response Theory to the Korean Nurses' Licensing Examination

2005 ◽  
Vol 2 ◽  
pp. 23 ◽  
Author(s):  
Geum-Hee Jeong ◽  
Mi Kyoung Yim
Keyword(s):  
Rasch Model ◽  
Logistic Model ◽  
Goodness Of Fit ◽  
Item Analysis ◽  
Classical Test Theory ◽  
Test Theory ◽  
Classical Test ◽  
Licensing Examination ◽  
The Rasch Model ◽  
Two Parameter

To test the applicability of item response theory (IRT) to the Korean Nurses' Licensing Examination (KNLE), item analysis was performed after testing the unidimensionality and goodness-of-fit. The results were compared with those based on classical test theory. The results of the 330-item KNLE administered to 12,024 examinees in January 2004 were analyzed. Unidimensionality was tested using DETECT and the goodness-of-fit was tested using WINSTEPS for the Rasch model and Bilog-MG for the two-parameter logistic model. Item analysis and ability estimation were done using WINSTEPS. Using DETECT, Dmax ranged from 0.1 to 0.23 for each subject. The mean square value of the infit and outfit values of all items using WINSTEPS ranged from 0.1 to 1.5, except for one item in pediatric nursing, which scored 1.53. Of the 330 items, 218 (42.7%) were misfit using the two-parameter logistic model of Bilog-MG. The correlation coefficients between the difficulty parameter using the Rasch model and the difficulty index from classical test theory ranged from 0.9039 to 0.9699. The correlation between the ability parameter using the Rasch model and the total score from classical test theory ranged from 0.9776 to 0.9984. Therefore, the results of the KNLE fit unidimensionality and goodness-of-fit for the Rasch model. The KNLE should be a good sample for analysis according to the IRT Rasch model, so further research using IRT is possible.

scholarly journals Analisis Pengetahuan Matematika, Didaktika, dan Teknologi Calon Guru Sekolah Dasar Menggunakan Rasch Model

2021 ◽  
Vol 10 (3) ◽  
pp. 345-356
Author(s):  
Zetra Hainul Putra ◽  
Neni Hermita ◽  
Jesi Alexander Alim
Keyword(s):  
Item Response Theory ◽  
Item Response ◽  
Rasch Model ◽  
Classical Test Theory ◽  
Technological Knowledge ◽  
Test Theory ◽  
Response Theory ◽  
Classical Test ◽  
Didactic Knowledge ◽  
The Rasch Model

AbstrakPengetahuan matematika, didaktika, dan teknologi yang harus dimilik oleh calon guru dan guru, sebelumnya diukur dengan metode classical test theory (CTT), namun memiliki keterbatasan. Penelitian bertujuan mengetahui pengetahuan tersebut dengan metode yang diprekdiksi lebih akurat yaitu item response theory (IRT) dari rasch model. Metode penelitian meliputi survey pengetahuan matematika dan didaktika serta survey pengetahuan teknologi menggunakan angket. Subjek penelitian yaitu 38 mahasiswa tahun kedua calon guru sekolah dasar dari sebuah institusi pendidikan guru sekolah dasar di Pekanbaru. Hasil studi menunjukkan calon guru sekolah dasar memiliki pengetahuan matematika dan teknologi cukup baik, namun pengetahuan didaktika sangat rendah. Analisis pengetahuan calon guru menggunkan rasch model mampu memberikan gambaran tingkat kesulitan soal/item yang diberikan, tingkat pengetahuan responden, dan interaksi antar responden dan antar soal. Institusi pendidikan perlu mengembangkan pembelajaran bagi calon guru sekolah dasar yang mampu mendukung penguatan dan pengembangan pengetahuan matematika, didaktika, dan teknologi. Prospective Elementary Teachers’ Mathematical, Didactic, and Technological Knowledge Using Rasch Model Analysis AbstractKnowledge of mathematics, didactics, and technology that must be owned by pre-service and in-service teachers, previously measured by the classical test theory (CTT), but has limitations. Study aims to determine that knowledge with a more accurate prediction method using the item response theory (IRT) from the Rasch model. Study used a survey of mathematical and didactic knowledge, and a survey of technological knowledge with a scale. The subjects were 38 second-year students from an elementary school teacher education institution in Pekanbaru. This study indicates that pre-service teachers have relatively sufficient mathematics and technological knowledge, but they were lack of didactic knowledge. Analysis of pre-service teachers’ knowledge using the Rasch model can provide an overview of the difficulty level of the items given, the level of pre-service teachers’ knowledge, and the interaction between respondents and between questions. Institutions need to develop learning instruction for pre-service teachers that can support the strengthening and development of their mathematical, didactic, and technological knowledge.

scholarly journals Uji Kualitas Uji Dokumentasi Terapi Menggunakan Teori Tes Klasik (CTT) dan Model Rasch

2018 ◽  
Vol 6 (2) ◽  
Author(s):  
Reza Nur a.
Keyword(s):  
Rasch Model ◽  
Classical Test Theory ◽  
Test Theory ◽  
Probability Sampling ◽  
Purposive Sampling ◽  
Classical Test

Mahasiswa okupasi terapi semester III Program Vokasi Kedokteran Universitas Indonesia harus memiliki kompetensi dokumentasi okupasi untuk menggali informasi yang ada pada pasien dan menentukan tujuan dari terapi yang akan diberikan untuk menjadi seorang okupasi terapis. Mata kuliah yang diajarkan belum dapat mengevaluasi dan menganalisa kompetensi mahasiswa terkait dokumentasi okupasi terapi. Masalah tersebut dianalisa menggunakan keilmuan psikologi (psikometrika) berupa Classical Test Rheory (CTT) dan Rasch Mode. Responden penelitian berjumlah 200 orang dengan alat ukur pola respon jawaban beserta kunci jawaban dari soal UAS Dokumentasi Okupasi, Vokasi Universitas Indonesia tahun 2015, 2016 dan 2017 yang terdiri dari 40 item (35 item pilihan ganda dan 5 item essay). Metode yang digunakan adalah non-probability sampling dengan teknik purposive sampling. Penelitian ini menunjukkan bahwa tingkat kesulitan item dokumentasi okupasi terapi yang tidak sebanding antara Classical Test Theory (CTT) dan Rasch Model dan Rasch Model berkaitan dengan hirarki pembelajaran mata kuliah dokumentasi okupasi terapi.

Sign in / Sign up

Export Citation Format

Share Document