Using Interval Likelihood Ratios in Gated Screening: A Direct Replication Study

This direct replication study compared the use of dichotomized likelihood ratios and interval likelihood ratios, derived using a prior sample of students, for predicting math risk in middle school. Data from the prior year state test and the Measures of Academic Progress were analyzed to evaluate differences in the efficiency and diagnostic accuracy of gated screening decisions. Post-test probabilities were interpreted using a threshold decision-making model to classify student risk during screening. Using interval likelihood ratios led to fewer students requiring additional testing after the first gate. But, when interval likelihood ratios were used, three tests were required to classify sixth and seventh grade students as at-risk or not at-risk. Only two tests were needed to classify students as at-risk or not at-risk when dichotomized likelihood ratios were used. Acceptable sensitivity and specificity estimates were obtained, regardless of the type of likelihood ratios used to estimate post-test probabilities. When predicting academic risk, interval likelihood ratios may be best reserved for situations where at least three successive tests are available to be used in a gated screening model.

Investigating school children’s behavioral and emotional problems using pediatric symptoms checklist-17 in a structural equation modeling framework

The Pediatric Symptom Checklist-17 was originally used in primary care settings with parents to identify their children’s behavioral and emotional problems, but there has been some research supporting use of this scale in school settings. This study examined: (a) the factor structure and measurement invariance of the teacher-rated Pediatric Symptom Checklist-17 and (b) complex relationships among demographic characteristics, behavioral and emotional problems, and learning outcomes using structural equation modeling in elementary schools. A sample of 508 children in grades one and two were rated by their teachers with the Pediatric Symptom Checklist-17. Measures of Academic Progress test was utilized to measure participants’ learning outcomes in reading and math. The results confirmed a three-factor structure of the Pediatric Symptom Checklist-17 (internalizing problems, externalizing problems, and attention problems) and attested the measurement invariance across different demographic groups (i.e. gender, ethnicity, and grade levels). Boys were more likely to have severe attention problems which were associated with lower learning outcomes as seen by Measures of Academic Progress reading and math scores. Attention problems mediated the relationship between gender and learning outcomes. This study has implications for the use of the Pediatric Symptom Checklist-17 in school-based settings. Additionally, it highlights the potential relationships among gender, attention problems, and learning outcomes.

Introducing a Computer-Adaptive Testing System to a Small School District

A computer-adaptive test (CAT) is a relatively new type of technology in which a computer program “intelligently” selects and presents questions to examinees according to an evolving estimate of achievement and a prescribed test plan. A well written CAT can be expected to efficiently produce student achievement estimates that are more accurate and more meaningful than a typical teacher-generated paper and pencil (P&P) test with a similar number of questions. Although this method of testing sounds good in theory, many schools and districts are waiting for positive examples of practical applications and observable benefits before adopting a CAT. This chapter begins by describing the essential elements of meaningful measurement in education and the features of a typical CAT. Next, we describe the Measures of Academic Progress (MAP) system of the Northwest Evaluation Association (NWEA; 2004) and observations made during the introduction of this system into a small semirural school district. Finally, as independent observers, we provide a set of recommendations to help guide other districts as they consider the potentials of implementing a CAT system to guide instruction within their schools.

Introducing a Computer-Adaptive Testing System to a Small School District

A computer-adaptive test (CAT) is a relatively new type of technology in which a computer program “intelligently” selects and presents questions to examinees according to an evolving estimate of achievement and a prescribed test plan. A well written CAT can be expected to efficiently produce student achievement estimates that are more accurate and more meaningful than a typical teacher-generated paper and pencil (P&P) test with a similar number of questions. Although this method of testing sounds good in theory, many schools and districts are waiting for positive examples of practical applications and observable benefits before adopting a CAT. This chapter begins by describing the essential elements of meaningful measurement in education and the features of a typical CAT. Next, we describe the Measures of Academic Progress (MAP) system of the Northwest Evaluation Association (NWEA; 2004) and observations made during the introduction of this system into a small semirural school district. Finally, as independent observers, we provide a set of recommendations to help guide other districts as they consider the potentials of implementing a CAT system to guide instruction within their schools.