Assessing the Reliability of the OMERACT Juvenile Idiopathic Arthritis Magnetic Resonance Scoring System for Temporomandibular Joints (JAMRIS-TMJ)

Contrast-enhanced magnetic resonance imaging (MRI) remains the most comprehensive modality to assess juvenile idiopathic arthritis (JIA)-related inflammation and osteochondral damage in the temporomandibular joints (TMJ). This study tested the reliability of a new JIA MRI scoring system for TMJ (JAMRIS-TMJ) and the impact of variations in calibration and reader specialty. Thirty-one MRI exams of bilateral TMJs were scored independently using the JAMRIS-TMJ by 20 readers consisting of radiologists and non-radiologist clinicians in three reading groups, with or without a calibrating atlas and/or tutorial. The inter-reader reliability in the multidisciplinary cohort assessed by the generalizability coefficient was 0.61–0.67 for the inflammatory and 0.66–0.74 for the damage domain. The atlas and tutorial did not improve agreement within radiologists, but improved the agreement between radiologist and non-radiologist groups. Agreements between different calibration levels were 0.02 to 0.08 lower by the generalizability coefficient compared to agreement within calibration levels; agreement between specialty groups was 0.04 to 0.10 lower than within specialty groups. Averaging two radiologists raised the reliability above 0.8 for both domains. Therefore, the reliability of JAMRIS-TMJ was moderate-to-good depending on the presence of specialty and calibration differences. The atlas and tutorial are necessary to improve reliability when the reader cohort consists of multiple specialties.

A Short Note on Optimizing Cost-Generalizability via a Machine-Learning Approach

The costs of an objective structured clinical examination (OSCE) are of concern to health profession educators globally. As OSCEs are usually designed under generalizability theory (G-theory) framework, this article proposes a machine-learning-based approach to optimize the costs, while maintaining the minimum required generalizability coefficient, a reliability-like index in G-theory. The authors adopted G-theory parameters yielded from an OSCE hosted by a medical school, reproduced the generalizability coefficients to prepare for optimizing manipulations, applied simulated annealing algorithm to calculate the number of facet levels minimizing the associated costs, and conducted the analysis in various conditions via computer simulation. With a given generalizability coefficient, the proposed approach, virtually an instrument of decision-making supports, found the optimal solution for the OSCE such that the associated costs were minimized. The computer simulation results showed how the cost reductions varied with different levels of required generalizability coefficients. Machine learning–based approaches can be used in conjunction with psychometric modeling to help planning assessment tasks more scientifically. The proposed approach is easy to adopt into practice and customize in alignment with specific testing designs. While these results are encouraging, the possible pitfalls such as algorithmic convergences’ failure and inadequate cost assumptions should also be avoided.

Cognitive Flexibility in Schoolchild Through the Graphic Representation of Movement

Neuroconstructivism postulates the progressive complexity of mental representation over the course of cognitive development and the role of the graphic representation of movement in the transformation of mental schemas, cognitive flexibility, and representational complexity. This study aims to: (1) understand children’s resources in the drawing of movement (5–8 years); and (2) verify whether there are differences in the graphic representation of movement as an indicator of cognitive flexibility. The participants were N = 240 children aged 5–8 years; 1,440 drawings were collected representing 2,880 characters (both animate and inanimate) from six stories. The analysis consisted: (1) data quality control, using the kappa coefficient, and Generalizability Theory to test the instrument’s validity and reliability; (2) Multivariate General Analysis and Mixed Linear Analysis of the factors (age and stories); (3) Multivariate General Analysis of the graphic components: categories and microcategories, as well as the elements that make up the macrocategories: “Static,” “Indication,” and “Movement”; and (4) calculation of the generalizability coefficient (G-coefficient). The results show that: (a) age best explains variability, with a high effect size (η2 = 0.732) across all components (F = 153.445; p < 0.001), thus increasing its complexity and (b) at ages 6 and 7, “Indication” appears as a modulator of “Static” (age 5) toward “Movement” (age 8). The generalizability coefficient is optimal (0.995). It is concluded that changes in the initial graphic representation of movement may interactively transform mental representation, thus increasing cognitive flexibility and prompting teaching applications to optimize such changes.

Analysis of Successful Offensive Play Patterns by the Spanish Soccer Team

Victory is the ultimate aim in soccer and therefore when a team wins an elite European or world championship, attempts will invariably be made to emulate the winning team’s style of play. In this study, we performed an in‐depth analysis of play by the Spanish soccer team during the 2012 UEFA European Championship, where it was crowned champion. Using observational methodology and T‐pattern analysis, we identified hidden patterns of play that ended in a goal for the Spanish team. A generalizability coefficient (e2) of 0.986 demonstrated that the offensive patterns detected are robust and highly generalizable. These patterns were formed by technical actions consisting of ball control and pass, with alternations between short and long passes, in the central area of the rival pitch, with use of both wings to achieve width of play and prioritization of width over depth of play. We also found patterns showing that goals and shots at goal were made on a ball delivered from the opposite direction to the shot and were not preceded by a technical action.

Estimating the Students’ Skill in Reciting and Writing Alqur’an at Faculty of Tarbiyah and Teacher Training IAIN Sultan Amai Gorontalo

<p class="ABSTRACTJUDUL">The purpose of this study was to estimate the instrument of reading and writing capability of the Qur’an in the practicum program students of the Faculty of Tarbiyah and Teacher Training IAIN Sultan Amai Gorontalo. This type of research is quantitative with an analysis approach to the generalization theory variance through the G-Study concept with a multifacet design p x r x I three facet variations. The results of the study prove the instrument of reading and reading ability of the Koran was tested, namely the estimated coefficient of reliability of the combined scores of the reading and writing assessment of the Koran from 20 students who were rated by 4 rater of 10 items which were judged by the magnitude of generalizability coefficient value of 0.82749 . The magnitude of this value indicates that the true score of the assessment results is quite high compared to the minimum reliability criteria, which is 0.70 fulfilling the reliable criteria.</p><p class="ABSTRACTJUDUL"> </p><p class="ABSTRACTJUDUL"><strong>Abstrak</strong></p><p class="ABSTRACT">Tujuan penelitian ini untuk mengestimasi instrument kemampuan baca tulis alqur’an pada program praktikum baca tulis Alqur’an mahasiswa Fakultas Ilmu Tarbiyah dan Keguruan IAIN Sultan Amai Gorontalo. Jenis penelitian ini menggunakan kuantitatif dengan pendekatan analisis varians teori generalisibilitas melalui konsep G-Study dengan desain multifacet p x r x I tiga variasi facet. Hasil penelitian membuktikan instrumen penilaian kemampuan baca tulis al-Qur’an teruji, yakni estimasi koefisien reliabilitas skor gabungan penilaian kemampuan baca tulis Alqur’an dari 20 mahasiswa yang dinilai oleh 4 rater terhadap 10 item yang dinilai diperoleh besarnya nilai koefisien generalizabilitas sebesar 0,82749. Besaran nilai tersebut menunjukkan bahwa true score hasil penilaian cukup tinggi dibandingkan kriteria reliabilitas minimal yaitu 0,70 memenuhi kriteria reliabel.</p><strong><span lang="EN-US"><br /></span></strong>

The Reliability of 2-Station Clerkship Objective Structured Clinical Examinations in Isolation and in Aggregate

Background: Most medical schools in the United States report having a 5- to 10-station objective structured clinical examination (OSCE) at the end of the core clerkship phase of the curriculum to assess clinical skills. We set out to investigate an alternative OSCE structure in which each clerkship has a 2-station OSCE. This study looked to determine the reliability of clerkship OSCEs in isolation to inform composite clerkship grading, as well as the reliability in aggregate, as a potential alternative to an end-of-third-year examination. Design: Clerkship OSCE data from the 2017-2018 academic year were analyzed: the generalizability coefficient (ρ2) and index of dependability (φ) were calculated for clerkships in isolation and in aggregate using variance components analysis. Results: In all, 93 students completed all examinations. The average generalizability coefficient for the individual clerkships was .47. Most often, the largest variance component was the interaction between the student and the station, indicating inconsistency in the performance of students between the 2 stations. Aggregate clerkship OSCE analysis demonstrated good reliability for consistency (ρ2 = .80). About one-third (33.8%) of the variance can be attributed to students, 8.2% can be attributed to the student by clerkship interaction, and 42.6% can be attributed to the student by block interaction, indicating that students’ relative performances varied by block. Conclusions: Two-station clerkship OSCEs have poor to fair reliability, and this should inform the weighting of the composite clerkship grade. Aggregating data results in good reliability. The largest source of variance in the aggregate was student by block, suggesting testing over several blocks may have advantages compared with a single day examination.

Improving generalizability coefficient estimate accuracy: A way to incorporate auxiliary information

Initially proposed by Marcoulides and further expanded by Raykov and Marcoulides, a structural equation modeling approach can be used in generalizability theory estimation. This article examines the utility of incorporating auxiliary variables into the structural equation modeling approach when missing data is present. In particular, the authors assert that by adapting a saturated correlates model strategy to structural equation modeling generalizability theory models, one can reduce any biased effects caused by missingness. Traditional approaches such as an analysis of variance do not possess such a feature. This article provides detailed instructions for adding auxiliary variables into a structural equation modeling generalizability theory model, demonstrates the corresponding benefits of bias reduction in generalizability coefficient estimate via simulations, and discusses issues relevant to the proposed approach.

Development of Physical Therapy Practical Assessment System by Using Multisource Feedback

The purposes of the research were (1) to develop the physical therapy practical assessment system by using the multisource feedback (MSF) approach and (2) to investigate the effectiveness of the implementation of the developed physical therapy practical assessment system. The development of physical therapy practical assessment system by using MSF was determined by nine experts in physical therapy. Suitability and feasibility of the system for each sub-group were investigated. Five input factors, two process factors, two output factors, and two feedback factors were involved in the system. Level of suitability and feasibility of elements in each sub-group presented at high to the highest level. In system testing, 40 physical therapy students were participated. Raters consisted of clinical educators, students (self-assessment), friends (students who in the same practical group), and patients. Twice assessments during the period of clinical practice were performed. Data analysis for generalizability coefficient (G-coefficient) was evaluated by the EduG program. Quality of the system was evaluated 4 aspects including the utility, feasibility, propriety, and accuracy. These were calculated by mean () and standard deviation (SD). The values of G-coefficient for absolute and relative were 0.86 and 0.88, respectively. In addition, quality of the system showed value at high to the highest level in all aspects.

Reliability of a viva assessment of clinical reasoning in an Australian pre-professional osteopathy program assessed using generalizability theory

Clinical reasoning is situation-dependent and case-specific; therefore, assessments incorporating different patient presentations are warranted. The present study aimed to determine the reliability of a multi-station case-based viva assessment of clinical reasoning in an Australian pre-registration osteopathy program using generalizability theory. Students (from years 4 and 5) and examiners were recruited from the osteopathy program at Southern Cross University, Lismore, Australia. The study took place on a single day in the student teaching clinic. Examiners were trained before the examination. Students were allocated to 1 of 3 rounds consisting of 5 10-minute stations in an objective structured clinical examination-style. Generalizability analysis was used to explore the reliability of the examination. Fifteen students and 5 faculty members participated in the study. The examination produced a generalizability coefficient of 0.53, with 18 stations required to achieve a generalizability coefficient of 0.80. The reliability estimations were acceptable and the psychometric findings related to the marking rubric and overall scores were acceptable; however, further work is required in examiner training and ensuring consistent case difficulty to improve the reliability of the examination.

Measuring multiple neuromuscular activation using EMG – a generalizability analysis

Analysis of electromyography (EMG) data has been shown to be valuable in biomedical and clinical research. However, most analysis tools do not consider the non-linearity of EMG data or the synergistic effects of multiple neuromuscular activities. The SYNERGOS algorithm was developed to assess a single index based on non-linear analysis of multiple neuromuscular activation (MNA) of different muscles. This index has shown promising results in Parkinsonian gait, but it was yet to be explored whether the SYNERGOS index is generalizable. In this study, we evaluated generalizability of the SYNERGOS index over the course of several trials and over separate days with different walking speeds. Ten healthy adults aged from 18 to 40 years walked on a treadmill on two different days, while EMG data was collected from the upper and lower right leg. SYNERGOS indices were obtained and a generalizability analysis was conducted. The algorithm detected changes in MNA in response to altering gait speed and depicted a high generalizability coefficient (