Item Pool Development for the School-Aged Life Impact Burn Recovery Evaluation (SA- LIBRE5-12) Profile Computerized Adaptive Test (CAT): An Observer-Reported Outcome Assessment Measuring the Impact of Burn Injuries in School-Aged Children

Introduction The transition from early childhood to teen years (5-12) is a critical time of development, which can be made particularly challenging by a burn injury. Assessing post-burn recovery during these years is important for improving pediatric survivors’ development and health outcomes. Few validated burn-specific measures exist for this age group. The purpose of this study was to generate item pools that will be used to create a future computerized adaptive test (CAT) assessing post-burn recovery in school-aged children. Methods Item pool development was guided by the previously developed School-Aged Life Impact Burn Recovery Evaluation (SA-LIBRE5-12) Conceptual Framework. The item pool development process involved a systematic literature review, extraction of candidate items from existing legacy measures, iterative item review during expert consensus meetings, and parent cognitive interviews. Results The iterative item review with experts consisted of six rounds. A total of 10 parent cognitive interviews were conducted. The three broad themes of concern were items that needed 1) clarification, needed context or were vague, 2) age dependence and relevance, and 3) word choice. The cognitive interviews indicated that survey instructions, recall period, item stem, and response choices were interpretable by respondents. Final item pool based on parental feedback consist of 57, 81, and 60 items in Physical, Psychological, and Family and Social Functioning respectively. Conclusion Developed item pools (n=198) in three domains are consistent with the existing conceptual framework. The next step involves field-testing the item pool and calibration using item response theory to develop and validate the SA-LIBRE5-12 CAT Profile.

maat: An R Package for Multiple Administrations Adaptive Testing

Multiple Administrations Adaptive Testing (MAAT) is an extension of the shadow-test approach to CAT for the assessment framework involving multiple tests administered periodically throughout the year. The maat package utilizes multiple item pools vertically scaled across grades and multiple phases (stages) within each test administration, allowing for transitioning from an item pool to another as deemed necessary to further enhance the quality of assessment.

Development and Feasibility of the Headache-Related Light and Sound Sensitivity Inventories in Youth

Youth with chronic headache disorders often experience sensitivities to light and sound that trigger or exacerbate their headaches and contribute to functional disability. At present, there are no known validated measures for assessing these sensitivities and their impact on functioning in youth with chronic headaches. This pilot study sought to develop and assess the feasibility of measures of headache-related light and sounds sensitivities in youth with chronic headache disorders. The initial item pools were generated via an intensive literature review, an informal quality improvement project, and a panel of experts in chronic pain. Then, youth (n = 20) presenting for clinical evaluation of headaches completed the revised items as well as assessments of the measures’ feasibility and items’ understandability. A subset (n = 2) completed formal cognitive interviews as well. The resulting 20-item Headache-Related Light Sensitivity Inventory (HALSI) and 18-item Headache-Related Sound Sensitivity Inventory (HASSI) for youth assess headache-related sensory sensitivities, as well as related emotional and behavioral responses. Through the iterative incorporation of feedback, these measures appear to be feasible to administer and understandable tools for assessing light and sound sensitivity in youth with chronic headache disorders. Once they are empirically validated, they have the potential to serve as important tools for understanding the patient experience, developing interventions, and assessing treatment response.

Linking of Rasch-Scaled Tests: Consequences of Limited Item Pools and Model Misfit

In the context of item response theory (IRT), linking the scales of two measurement points is a prerequisite to examine a change in competence over time. In educational large-scale assessments, non-identical test forms sharing a number of anchor-items are frequently scaled and linked using two− or three-parametric item response models. However, if item pools are limited and/or sample sizes are small to medium, the sparser Rasch model is a suitable alternative regarding the precision of parameter estimation. As the Rasch model implies stricter assumptions about the response process, a violation of these assumptions may manifest as model misfit in form of item discrimination parameters empirically deviating from their fixed value of one. The present simulation study investigated the performance of four IRT linking methods—fixed parameter calibration, mean/mean linking, weighted mean/mean linking, and concurrent calibration—applied to Rasch-scaled data with a small item pool. Moreover, the number of anchor items required in the absence/presence of moderate model misfit was investigated in small to medium sample sizes. Effects on the link outcome were operationalized as bias, relative bias, and root mean square error of the estimated sample mean and variance of the latent variable. In the light of this limited context, concurrent calibration had substantial convergence issues, while the other methods resulted in an overall satisfying and similar parameter recovery—even in the presence of moderate model misfit. Our findings suggest that in case of model misfit, the share of anchor items should exceed 20% as is currently proposed in the literature. Future studies should further investigate the effects of anchor item composition regarding unbalanced model misfit.

The Optimal Setting of A/B Exam Papers without Item Pools: A Hybrid Approach of IRT and BGP

The administration of A/B exams usually involves the use of items. Issues arise when the pre-establishment of a question bank is necessary and the inconsistency in the knowledge points to be tested (in the two exams) reduces the exams ‘fairness’. These are critical for a large multi-teacher course wherein the teachers are changed such that the course and examination content are altered every few years. However, a fair test with randomly participating students should still be a guaranteed subject with no item pool. Through data-driven decision-making, this study collected data related to a term test for a compulsory general course for empirical assessments, pre-processed the data and used item response theory to statistically estimate the difficulty, discrimination and lower asymptotic for each item in the two exam papers. Binary goal programing was finally used to analyze and balance the fairness of A/B exams without an item pool. As a result, pairs of associated questions in the two exam papers were optimized in terms of their overall balance in three dimensions (as the goals) through the paired exchanges of items. These exam papers guarantee their consistency (in the tested knowledge points) and also ensure the fairness of the term test (a key psychological factor that motivates continued studies). Such an application is novel as the teacher(s) did not have a pre-set question bank and could formulate the fairest strategy for the A/B exam papers. The model can be employed to address similar teaching practice issues.

776 A New Outcome Metric for Young Children with Burns: Item Pool Development for the Preschool Life Impact Burn Recovery Evaluation (LIBRE) Computer Adaptive Test (CAT) Profile

Introduction The assessment of recovery from burn injury is critical to improving pediatric health. However, there is a lack of brief, burn-specific measures to assess preschool-aged burn survivors’ health outcomes. We developed items pools for a new, parent-reported computer adaptive test based assessment of preschool-aged children’s burn outcomes. Methods Initial item pools were informed by the Preschool LIBRE Conceptual Model based on the International Classification for Children (ICF) and the Burn Outcomes Questionnaires (BOQ 0–5) and generated from a review of existing instruments assessing health and developmental outcomes in children 1–5 years. Candidate items underwent a review process to bin items together based on the underlying construct they assessed. Items were then winnowed down and further refined based on clinical expert consensus meetings. The winnowing process focused on ensuring items assessed abilities and behaviors that could be reliably determined by parent report, were age-relevant, and focused on health and developmental concepts that are important in the assessment of children’s burn recovery. To ensure item quality, we conducted parent cognitive interviews. In final reviews, we standardized item recall periods, response options, verb tense, and literacy demands. Results A total of 9,509 items were identified and extracted. Four item pool domains were established: communication and language development (42 items), physical functioning (53 items), psychological functioning (56 items), and social functioning (37 items). The respective item pools aim to assess: children’s ability to receive meaning and produce language; children’s gross and fine motor abilities; children’s emotions and behavior (internalizing and externalizing behaviors, dysregulation, toileting, response to trauma and resilience); and children’s social participation and abilities. Recall period and response options were refined for consistency for all 188 final items. Conclusions Four item pools were developed assessing four domains 1) communication and language development, 2) physical functioning, 3) psychological functioning and 4) social functioning for preschool-aged children. The item pools are currently being field-tested for the calibration and validation of the Preschool LIBRE CAT Profile. Applicability of Research to Practice This work is relevant to researchers and practitioners evaluating the effects of burn injury on preschool-aged children’s health and developmental outcomes.

The Optimal Item Pool Design in Multistage Computerized Adaptive Tests With the p-Optimality Method

The present study extended the p-optimality method to the multistage computerized adaptive test (MST) context in developing optimal item pools to support different MST panel designs under different test configurations. Using the Rasch model, simulated optimal item pools were generated with and without practical constraints of exposure control. A total number of 72 simulated optimal item pools were generated and evaluated by an overall sample and conditional sample using various statistical measures. Results showed that the optimal item pools built with the p-optimality method provide sufficient measurement accuracy under all simulated MST panel designs. Exposure control affected the item pool size, but not the item distributions and item pool characteristics. This study demonstrated that the p-optimality method can adapt to MST item pool design, facilitate the MST assembly process, and improve its scoring accuracy.

The Psychological Characteristics of Situations: Towards an Integrated Taxonomy

Much current research on situations, instead of attending to single situation cues or abstract classes of situations, focuses on characteristics which capture the psychological meaning and interpretation of a situation. This focus allows a differential psychology of situations where any situation can be described and compared with a set of continuous dimensions (much like how persons can be described by traits). We provide an overview of extant taxonomies of situation characteristics and highlight their convergences. Several taxonomies—independently developed from different research teams with different item pools, samples, and data-analytic methods—have provided psychometrically validated measurement tools which make it possible to associate the (scales of the) taxonomies with one another. Upon examining the conceptual and empirical overlaps between those taxonomies, six replicable domains of situation characteristics can be identified: threat, stress, tasks, processing, fun, and mundane. These domains bear in content a striking resemblance to the Big Five or HEXACO dimensions of personality traits. We sketch how theory, methods, and research on situations may make further progress, especially regarding the goal of crafting an integrative and inclusive taxonomy of situation characteristics.

Introducing Item Pool Visualization: A method for investigation of concepts in self-reports and psychometric tests

In this article, we introduce Item Pool Visualization. It is an illustration system that locates items and item pools (scales) from multiple psychological instruments regarding their commonality and distinguishability along several dimensions of nested radar charts. The application of Item Pool Visualization creates illustrations that represent different item pools by different circles that do not overlap. Item Pool Visualization illustrates a comparison of different structural equation models that are estimated with the same data. It combines the advantages of general and correlated factor models when evaluating psychological instruments. Furthermore, in contrast to other visualization methods, Item Pool Visualization provides an empirically driven categorization of psychological constructs and their subconcepts (facets) that is suited to provide professionals with help in comparing psychometric constructs, questionnaires, and selecting tests.