Development and evaluation of a HOTS-based test for matrix topic: A classical test and item response theory

[English]: This research aims to develop and evaluate a higher-order thinking skill (HOTS)-based test for a matrix topic. The development was carried out in two stages; items development and validation. The first stage was to review relevant literature about HOTS, design the test items, have experts review, and try out the items. Fifty-one upper secondary school students were involved in the tryout. In the second stage, results of the tryout were validated referring to the classical test and item response theory, including items characteristics, validity and reliability, items discrimination, and difficulty levels. The validation resulted in five valid test items (r1=0,54; r2=0,88; r3=0,72; r4=0,78; r5=0,82). The developed test represents the topic, fulfills HOTS criteria, is reliable rα=0,85, can differentiate students with higher-order thinking, and has varied difficulty levels. [Bahasa]: Penelitian ini bertujuan untuk mengembangkan dan mengevaluasi soal tes berbasis keterampilan berpikir tingkat tinggi (HOTS) pada materi matriks. Pengembangan instrumen tes melalui dua tahap, yaitu pengembangan draf soal dan validasi. Pada tahap pertama, dilakukan kajian literatur yang relevan, penyusunan rencana butir soal, evaluasi butir soal yang diusulkan, dan uji coba draf butir soal. Sebanyak 51 siswa sekolah menengah dilibatkan pada tahapan uji coba. Pada tahap validasi, dilakukan analisis menggunakan teori tes klasik dan teori respon butir mencakup: karakterisasi, validitas dan reliabilitas, uji daya beda, dan tingkat kesulitan soal. Penelitian ini menghasilkan lima butir soal yang valid (r1=0,54; r2=0,88; r3=0,72; r4=0,78; r5=0,82). Tes yang dikembangkan mewakili materi matriks, memenuhi kriteria HOTS, dapat diandalkan dengan nilai reliabilitas tes sebesar rα=0,85, dapat membedakan siswa yang memiliki kemampuan berpikir tingkat tinggi, dan memiliki keragaman tingkat kesulitan.

Karakteristik Butir Soal Penilaian Akhir Semester Mata Pelajaran Sejarah Kelas XI SMA Negeri 1 Pangkalpinang

This research is a study of quantitative descriptive. The purpose of this research is to describe the characteristics of final semester exam items for grade XI in the History subject at SMA Negeri 1 Pangkalpinang using the classical test theory approach. The research of the subject was 138 students of class XI in Social Sciences Major. The result of the research shows that final exam questions in the history subject class XI of SMA Negeri 1 Pangkalpinang are proper to use. This shows that from the validity of the items which there are 39 items of questions (97.5%) which are proven empirically valid with a 0.818 reliability coefficient. Other than that, there are 27 items of questions (67,5%) that can fulfill the criteria for the difficulty level, distinguishing power, and distractor function so it can be used directly to measure the student's ability without correction. While 12 items of questions (30%) need to be fixed and 1 item of question (2,5%) is declared to be invalid so it can't be used to measure the student's ability in History Subject. Permasalahan yang melatarbelakangi penelitian ini adalah pengembangan soal penilaian akhir semester mata pelajaran sejarah yang tidak melalui tahapan analisis butir soal sehingga kualitas butir soal tidak diketahui. Penelitian ini merupakan penelitian deskriptif kuantitatif. Tujuan penelitian ini adalah untuk mendeskripsikan karakteristik butir soal penilaian akhir semester mata pelajaran sejarah kelas XI SMA Negeri 1 Pangkalpinang menggunakan pendekatan teori tes klasik. Subjek penelitian berjumlah 138 peserta didik kelas XI jurusan IPS. Hasil penelitian menunjukkan bahwa soal PAS mata pelajaran sejarah kelas XI SMA Negeri 1 Pangkalpinang telah layak digunakan. Hal ini dibuktikan dari validitas butir soal yang mana terdapat 39 butir soal (97,5%) terbukti valid secara empirik dengan koefisien reliabilitas sebesar 0,818. Selain itu terdapat 27 butir soal (67,5%) yang memenuhi kriteria tingkat kesukaran, daya beda, dan keberfungsian distraktor sehingga dapat digunakan langsung untuk mengukur kemampuan peserta didik tanpa perbaikan. Sedangkan sebanyak 12 butir soal (30%) perlu dilakukan perbaikan dan 1 butir soal (2,5%) dinyatakan gugur sehingga tidak dapat digunakan untuk mengukur kemampuan peserta didik pada mata pelajaran sejarah.

Validating Insomnia Severity Index (ISI) in a Bangladeshi Population: Using Classical Test Theory and Rasch Analysis

The COVID-19 outbreak is associated with sleep problems and mental health issues among individuals. Therefore, there is a need to assess sleep efficiency during this tough period. Unfortunately, the commonly used instrument on insomnia severity—the Insomnia Severity Index (ISI)—has never been translated and validated among Bangladeshis. Additionally, the ISI has never been validated during a major protracted disaster (such as the COVID-19 outbreak) when individuals encounter mental health problems. The present study aimed to translate the ISI into Bangla language (ISI-Bangla) and validate its psychometric properties. First, the linguistic validity of the ISI-Bangla was established. Then, 9790 Bangladeshis (mean age = 26.7 years; SD = 8.5; 5489 [56.1%] males) completed the Bangla versions of the following questionnaires: ISI, Fear of COVID-19 Scale (FCV-19S), and Patient Health Questionnaire-9 (PHQ-9). All the participants also answered an item on suicidal ideation. Classical test theory and Rasch analyses were conducted to evaluate the psychometric properties of the ISI-Bangla. Both classical test theory and Rasch analyses support a one-factor structure for the ISI-Bangla. Moreover, no substantial differential item functioning was observed across different subgroups (gender, depression status (determined using PHQ-9), and suicidal ideation). Additionally, concurrent validity of the ISI-Bangla was supported by significant and moderate correlations with FCV-19S and PHQ-9; known-group validity was established by the significant difference of the ISI-Bangla scores between participants who experienced suicidal ideation and those without. The present psychometric validation conducted during the COVID-19 outbreak suggests that the ISI-Bangla is a promising and operationally adequate instrument to assess insomnia in Bangladeshis.

Integrating informative hypotheses into the EffectLiteR framework

Using the EffectLiteR framework, researchers can test classical null hypotheses about effects of interest via Wald and F-tests, while taking into account the stochastic nature of group sizes. This paper aims at extending EffectLiteR to test informative hypotheses, assuming for example that the average effect of a new treatment is greater than the average effect of an old treatment, which in turn is greater than zero. We present a simulated data example to show two methodological novelties. First, we illustrate how to use the Fbar- and generalized linear Wald test to assess informative hypotheses. While the classical test did not reach significance, the informative test correctly rejected the null hypothesis, indicating the need to take into account the order of the treatment groups. Second, we demonstrate how to account for stochastic group sizes in informative hypotheses using the generalized non-linear Wald statistic. The paper concludes with a short data example.

Quality of life and self-stigma of schizophrenia patient’s caregiver tool: Development and validation using classical test theory and Rasch analysis

Background: Providing care for schizophrenia patients is complex, and it requires dealing with various psychosocial burdens.Aim: To develop and validate a tool that measures the quality of life and self-stigma (SS) of the schizophrenia patient’s caregiver (QLSSoSPC).Setting: Outpatient psychiatric services clinics in Saudi Arabia.Methods: The current study used a methodological cross-sectional design. A sample of 205 schizophrenia patients’ caregivers was recruited by using a convenient sampling method. Classical Test Theory and Rasch Analysis approaches were used.Results: The developed tool has proven acceptable level of reliability and validity. The analysis confirmed seven-factor structure accounted for 74.4% of the total variance. Cronbach’s reliability statistics for the developed tool were satisfactory and ranged from 0.80 to 0.91.Conclusion: The psychometric properties of the QLSSoSPC tool supported its prospective use and allowing us to recommend the implementation of the tool on behalf of clinical and research purposes.