A Generalizability Analysis of the Meaning in Life Questionnaire for Chinese Adolescents

The level of meaning in life not only affects the physical health of individuals, but also is closely related to their mental health. At present, many self-reported questionnaires are being used to measure the meaning in life of Chinese adolescents. Using the multivariate generalizability theory, this study investigated the psychometric properties and the internal structure of the Meaning in Life Questionnaires (MLQs), the most widely used questionnaire for assessing the level of meaning in life of Chinese adolescents. The data were sample of 1,951 junior high school students from Guizhou, China. Multivariate random measurement mode p × i° is the primary analytic approach. Results showed that the generalizability coefficient and dependability index of the scale were 0.86 and 0.85, respectively. The generalizability coefficients of presence of meaning and search for meaning were 0.76 and 0.85, respectively, and the dependability indexes were 0.75 and 0.85 for MLQ-P and MLQ-S, respectively. The design of each factor for MLQ is reasonable in terms of score ratio and the number of projects. In brief, the reliability and factor structure of the scale are satisfactory.

Extended Multivariate Generalizability Theory With Complex Design Structures

This article extends multivariate generalizability theory (MGT) to tests with different random-effects designs for each level of a fixed facet. There are numerous situations in which the design of a test and the resulting data structure are not definable by a single design. One example is mixed-format tests that are composed of multiple-choice and free-response items, with the latter involving variability attributable to both items and raters. In this case, two distinct designs are needed to fully characterize the design and capture potential sources of error associated with each item format. Another example involves tests containing both testlets and one or more stand-alone sets of items. Testlet effects need to be taken into account for the testlet-based items, but not the stand-alone sets of items. This article presents an extension of MGT that faithfully models such complex test designs, along with two real-data examples. Among other things, these examples illustrate that estimates of error variance, error–tolerance ratios, and reliability-like coefficients can be biased if there is a mismatch between the user-specified universe of generalization and the complex nature of the test.

Initial Validation Evidence for Clinical Case Presentations by Student Pharmacists

Objective: There is a paucity of validation evidence for assessing clinical case-presentations by Doctor of Pharmacy (PharmD) students. Within Kane’s Framework for Validation, evidence for inferences of scoring and generalization should be generated first. Thus, our objectives were to characterize and improve scoring, as well as build initial generalization evidence, in order to provide validation evidence for performance-based assessment of clinical case-presentations. Design: Third-year PharmD students worked up patient-cases from a local hospital. Students orally presented and defended their therapeutic care-plan to pharmacist preceptors (evaluators) and fellow students. Evaluators scored each presentation using an 11-item instrument with a 6-point rating-scale. In addition, evaluators scored a global-item with a 4-point rating-scale. Rasch Measurement was used for scoring analysis, while Generalizability Theory was used for generalization analysis. Findings: Thirty students each presented five cases that were evaluated by 15 preceptors using an 11-item instrument. Using Rasch Measurement, the 11-item instrument’s 6-point rating-scale did not work; it only worked once collapsed to a 4-point rating-scale. This revised 11-item instrument also showed redundancy. Alternatively, the global-item performed reasonably on its own. Using multivariate Generalizability Theory, the g-coefficient (reliability) for the series of five case-presentations was 0.76 with the 11-item instrument, and 0.78 with the global-item. Reliability was largely dependent on multiple case-presentations and, to a lesser extent, the number of evaluators per case-presentation. Conclusions: Our pilot results confirm that scoring should be simple (scale and instrument). More specifically, the longer 11-item instrument measured but had redundancy, whereas the single global-item provided measurement over multiple case-presentations. Further, acceptable reliability can be balanced between more/fewer case-presentations and using more/fewer evaluators.

Indices of Subscore Utility for Individuals and Subgroups Based on Multivariate Generalizability Theory

Conventional methods for evaluating the utility of subscores rely on traditional indices of reliability and on correlations among subscores. One limitation of correlational methods is that they do not explicitly consider variation in subtest means. An exception is an index of score profile reliability designated as [Formula: see text], which quantifies the ratio of true score profile variance to observed score profile variance. [Formula: see text] has been shown to be more sensitive than correlational methods to group differences in score profile utility. However, it is a group average, representing the expected value over a population of examinees. Just as score reliability varies across individuals and subgroups, one can expect that the reliability of score profiles will vary across examinees. This article proposes two conditional indices of score profile utility grounded in multivariate generalizability theory. The first is based on the ratio of observed profile variance to the profile variance that can be attributed to random error. The second quantifies the proportion of observed variability in a score profile that can be attributed to true score profile variance. The article describes the indices, illustrates their use with two empirical examples, and evaluates their properties with simulated data. The results suggest that the proposed estimators of profile error variance are consistent with the known error in simulated score profiles and that they provide information beyond that provided by traditional measures of subscore utility. The simulation study suggests that artificially large values of the indices could occur for about 5% to 8% of examinees. The article concludes by suggesting possible applications of the indices and discusses avenues for further research.

A modification of the Schutte Emotional Intelligence Scale using multivariate generalizability theory

We modified the Schutte Emotional Intelligence Scale (EIS) using multivariate generalizability theory (MGT), a measurement system in which the focus is on multiple universe scores within a specific generalization universe. Application of MGT comprises conducting a generalizability study to analyze and estimate the associations among various factors, the number of factors, and the initial measurement design process, as well as a decision study to explore changes in the generalized coefficient and reliability indices when the number of items in the measure is changed. We conducted a generalizability study with 210 students from Shanghai and Anhui Province China, and performed item analysis and exploratory factor analysis, then modified the scale using MGT to create a modified edition of the EIS and tested item and factor reliability and validity with another 1,339 students from Shanghai and Anhui Province. Results revealed better reliability and validity than the original EIS, with each factor making a more balanced contribution to the total variance. The modified Chinese EIS will be useful for future emotional intelligence research.

The Use of Multivariate Generalizability Theory to Evaluate the Quality of Subscores

Conventional methods for evaluating the utility of subscores rely on reliability and correlation coefficients. However, correlations can overlook a notable source of variability: variation in subtest means/difficulties. Brennan introduced a reliability index for score profiles based on multivariate generalizability theory, designated as [Formula: see text], which is sensitive to variation in subtest difficulty. However, there has been little, if any, research evaluating the properties of this index. A series of simulation experiments, as well as analyses of real data, were conducted to investigate [Formula: see text] under various conditions of subtest reliability, subtest correlations, and variability in subtest means. Three pilot studies evaluated [Formula: see text] in the context of a single group of examinees. Results of the pilots indicated that [Formula: see text] indices were typically low; across the 108 experimental conditions, [Formula: see text] ranged from .23 to .86, with an overall mean of 0.63. The findings were consistent with previous research, indicating that subscores often do not have interpretive value. Importantly, there were many conditions for which the correlation-based method known as proportion reduction in mean-square error (PRMSE; Haberman, 2006) indicated that subscores were worth reporting, but for which values of [Formula: see text] fell into the .50s, .60s, and .70s. The main study investigated [Formula: see text] within the context of score profiles for examinee subgroups. Again, not only [Formula: see text] indices were generally low, but it was also found that [Formula: see text] can be sensitive to subgroup differences when PRMSE is not. Analyses of real data and subsequent discussion address how [Formula: see text] can supplement PRMSE for characterizing the quality of subscores.