scholarly journals Bayesian modeling of item heterogeneity in dichotomous recognition memory data and prospects for computerized adaptive testing

2021 ◽  
Author(s):  
Jeremie Güsten ◽  
David Berron ◽  
Emrah Duzel ◽  
Gabriel Ziegler
Keyword(s):  
Recognition Memory ◽  
Bayesian Modeling ◽  
Computerized Adaptive Testing ◽  
Item Difficulty ◽  
Face Validity ◽  
High Threshold ◽  
Adaptive Procedure ◽  
Modeling Framework ◽  
Ability Estimates ◽  
Out Of Sample Prediction

Most current models of recognition memory fail to separately model item and person heterogeneity which makes it difficult to assess ability at the latent construct level and prevents the administration of adaptive tests. Here we propose to employ a General Condorcet Model for Recognition (GCMR) in order to estimate ability, response bias and item difficulty in dichotomous recognition memory tasks. Using a Bayesian modeling framework and MCMC inference, we perform 3 separate validation studies comparing GCMR to the Rasch model from IRT and the 2-High-Threshold (2HT) recognition model. First, two simulations demonstrate that recovery of GCMR ability estimates with varying sparsity and test difficulty is more robust and thatestimates improve from the two other models under common test scenarios. Then, using a real dataset, face validity is confirmed by replicating previous findings of general and domain-specific age effects (Güsten et al., 2021). Using cross-validation we show better out-of-sample prediction for the GCMR as compared to Rasch and 2HT model. Finally, an adaptive test using the GCMR is simulated, showing that the test length necessary to obtain reliable ability estimates can be significantly reduced compared to a non-adaptive procedure. The GCMR allows to model trial-by-trial performance and to increase the efficiency and reliability of recognition memory assessments.

scholarly journals Cross-Cultural Adaptation and Psychometric Properties of Osteoporosis Knowledge Tool-Arabic Version Among Iraqi Population

2019 ◽  
Vol 13 (1) ◽  
pp. 30-38
Author(s):  
Shaymaa Abdalwahed Abdulameer ◽  
Mohanad Naji Sahib
Keyword(s):  
Public Health ◽  
General Population ◽  
Item Difficulty ◽  
Discriminatory Power ◽  
Face Validity ◽  
Arabic Version ◽  
Index Point ◽  
Point Biserial Correlation ◽  
Osteoporosis Knowledge ◽  
Difficulty Index

Background:Osteoporosis is a major public health problem as the majority of people are not aware of the disease until the complications occur.Objective:The aims of this study were to validate Osteoporosis Knowledge Tool (OKT-A) Arabic version and to assess the osteoporosis knowledge among Iraqi general population.Methods:A descriptive, cross-sectional study was carried out in the city of Baghdad with a random cluster sampling method from the community. Forward–backward-forward translation method was used to translate the OKT questionnaire from English into Arabic language. The psychometric assessment process includes: face validity, reliability (Cronbach’s alpha and test-retest), item difficulty index, point biserial correlation and discriminatory power.Results:The results showed good face validity. The Cronbach’s alpha and Pearson correlation coefficient of the test re-test reliability were 0.775 and 0.412, respectively. Item difficulty index, point biserial correlation ranges and discriminatory power were 0.105 to 0.852, 0.105 to 0.445 and 0.933, respectively. These results demonstrated that OKT-A was a reliable and stable tool. The results showed low OKT-A scores 11.50±3.958. Furthermore, the OKT-A scores and its subscales were less than 50%. In addition, there were significant differences between the following independent variables in relation to total OKT-A scores: educational level, do you have osteoporosis or ever heard about osteoporosis. Moreover, there was a significant association between ever heard about osteoporosis groups and the OKT-A knowledge levels.Conclusion:This study showed good validity and reliability of OKT-A tool among Arabic general population. In addition, the results showed an urgent need for implementing an educational programme and should be a public health practice to increase the knowledge toward osteoporosis and its related risk factor.

scholarly journals Development and validation of a multiple-choice question-based delirium care knowledge quiz for critical care nurses

2021 ◽  
Author(s):  
Mu-Hsing Ho
Keyword(s):  
Critical Care ◽  
Construct Validity ◽  
Internal Consistency ◽  
Item Difficulty ◽  
Item Analysis ◽  
Intraclass Correlation ◽  
Multiple Choice ◽  
Critical Care Nurses ◽  
Face Validity ◽  
Development And Validation

ABSTRACTAimsTo develop and psychometrically test a multiple choice questions (MCQs)-based quiz of delirium care knowledge for critical care nurses.DesignInstrument development and psychometric evaluation study.MethodsThe development and validation process including two phases. Phase I focused on the quiz development, conducted by the following steps: (1) generated initial 20-item pool; (2) examined content validity and (3) face validity; (4) conducted pilot testing, data were collected from 217 critical care nurses via online survey during 01 October to 07 November, 2020; (5) performed item analysis and eliminated items based on the item difficulty and discrimination indices. The MCQs quiz was finalised through the development process. Then, phases II emphasised the quiz validation, to estimate the internal consistency, split-half and test-retest reliability, and construct validity using parallel analysis with the exploratory factor analysis (EFA).ResultsA final 16-item MCQs quiz was emerged from the item analysis. The Kuder– Richardson Formula 20 coefficient for the overall quiz showed good internal consistency (0.85), and the intraclass correlation coefficient with a 30-day interval also indicated that the questionnaire had satisfactory stability (0.96). The EFA confirmed appropriate construct validity for the quiz, four factors could explain the total variance of 60.87%.ConclusionThis study developed the first MCQs quiz for delirium care knowledge and it is a reliable and valid tool that can be implemented to assess the level of delirium care knowledge.ImpactThis study offers an evidence-based quiz designed for future research and education purposes in delirium care that has significant implications for knowledge test by using MCQs in clinical practice.

scholarly journals A Dynamic Stratification Method for Improving Trait Estimation in Computerized Adaptive Testing Under Item Exposure Control

2019 ◽  
Vol 44 (3) ◽  
pp. 182-196
Author(s):  
Jyun-Hong Chen ◽  
Hsiu-Yi Chao ◽  
Shu-Ying Chen
Keyword(s):  
Computerized Adaptive Testing ◽  
Item Difficulty ◽  
Adaptive Testing ◽  
Item Selection ◽  
Exposure Control ◽  
Item Exposure ◽  
Stratification Method ◽  
High Discrimination ◽  
Item Exposure Control ◽  
Trait Estimation

When computerized adaptive testing (CAT) is under stringent item exposure control, the precision of trait estimation will substantially decrease. A new item selection method, the dynamic Stratification method based on Dominance Curves (SDC), which is aimed at improving trait estimation, is proposed to mitigate this problem. The objective function of the SDC in item selection is to maximize the sum of test information for all examinees rather than maximizing item information for individual examinees at a single-item administration, as in conventional CAT. To achieve this objective, the SDC uses dominance curves to stratify an item pool into strata with the number being equal to the test length to precisely and accurately increase the quality of the administered items as the test progresses, reducing the likelihood that a high-discrimination item will be administered to an examinee whose ability is not close to the item difficulty. Furthermore, the SDC incorporates a dynamic process for on-the-fly item–stratum adjustment to optimize the use of quality items. Simulation studies were conducted to investigate the performance of the SDC in CAT under item exposure control at different levels of severity. According to the results, the SDC can efficiently improve trait estimation in CAT through greater precision and more accurate trait estimation than those generated by other methods (e.g., the maximum Fisher information method) in most conditions.

Examining Word List Selection and Performance: An Explanatory Item Analysis of the CERAD Word List Learning Test

2021 ◽  
Author(s):  
William Goette
Keyword(s):  
Fixed Effects ◽  
Bayesian Model Averaging ◽  
Item Difficulty ◽  
Word List ◽  
Information Criteria ◽  
Modeling Framework ◽  
Assessment Protocol ◽  
Irt Models ◽  
List Learning ◽  
Learning Test

Objective: Develop and test an explanatory item response theory model (IRT) that examines properties of both the test (e.g., word order, learning over trials) and items (e.g., frequency of words in English) on the CERAD List Learning Test immediate recall trials.Methods: Item-level response data from 1050 participants (Mage=73.74 [SD=6.89], Medu=13.77 [SD=2.41]) in the Harmonized Cognitive Assessment Protocol were used to construct various IRT models. A Bayesian generalized (non-)linear multilevel modeling framework was utilized to specify the Rasch and two-parameter logistic (2PL) IRT models. Leave-one-out cross-validation information criteria and pseudo-Bayesian model averaging were used to compare models. Posterior predictive checks helped validate model performance in predicting data observations. Fixed effects for learning over trials, serial position of words, and 9 word properties of the words (obtained through the English Lexicon Project) were modeled for their effects on item properties.Results: A random person, random item 2PL model with an item-specific inter-trial learning effect (i.e., local dependency effect) provided the best fit of any of the models examined. Of the 9 word traits examined, only 4 has highly probable effects on item difficulty such that words became harder to learn with increasing frequency in English, average age of acquisition, and concreteness and lower levels of body-object integration.Conclusions: Results support that memory performance depends on more than repetition of words across trials. The finding that word traits affect difficulty and predict learning raise interesting potentials for test translation, equating word lists, and extending test interpretation to more nuanced semantic deficits.

Computerized Adaptive Testing: Tracking Candidate Response Patterns

1995 ◽  
Vol 13 (2) ◽  
pp. 151-162 ◽  
Author(s):  
Mary E. Lunz ◽  
Betty Bergstrom
Keyword(s):  
Computerized Adaptive Testing ◽  
Adaptive Testing ◽  
Response Patterns ◽  
Certification Examination ◽  
Adaptive Tests ◽  
Computerized Adaptive Test ◽  
Computerized Adaptive Tests ◽  
Ability Estimates ◽  
Adaptive Test ◽  
Candidate Performance

Computerized adaptive testing (CAT) uses a computer algorithm to construct and score the best possible individualized or tailored tests for each candidate. The computer also provides an absolute record of all responses and changes to responses, as well as their effects on candidate performance. The detail of the data from computerized adaptive tests makes it possible to track initial responses and response alterations, and their effect on candidate estimated ability measures, as well as the statistical performance of the examination. The purpose of this study was to track the effect of candidate response patterns on a computerized adaptive test. A ninety-item certification examination was divided into nine units of ten items each to track the pattern of initial responses and response alterations on ability estimates and test precision across the nine test units. The precision of the test was affected most by response alterations during early segments of the test. While generally, candidates benefit from altering responses, individual candidates showed different patterns of response alterations across test segments. Test precision is minimally affected, suggesting that the tailoring of CAT is minimally affected by response alterations.

Sign in / Sign up

Export Citation Format

Share Document