scholarly journals Accuracy of Automated Written Expression Curriculum-Based Measurement Scoring

2021 ◽  
pp. 082957352098775
Author(s):  
Sterett H. Mercer ◽  
Joanna E. Cannon ◽  
Bonita Squires ◽  
Yue Guo ◽  
Ella Pinco
Keyword(s):  
Progress Monitoring ◽  
Preliminary Evidence ◽  
Written Expression ◽  
Student Writing ◽  
Curriculum Based Measurement ◽  
Narrative Writing ◽  
Large Numbers ◽  
Community Based Organization ◽  
Highly Correlated ◽  
Total Words Written

We examined the extent to which automated written expression curriculum-based measurement (aWE-CBM) can be accurately used to computer score student writing samples for screening and progress monitoring. Students ( n = 174) with learning difficulties in Grades 1 to 12 who received 1:1 academic tutoring through a community-based organization completed narrative writing samples in the fall and spring across two academic years. The samples were evaluated using four automated and hand-calculated WE-CBM scoring metrics. Results indicated automated and hand-calculated scores were highly correlated at all four timepoints for counts of total words written ( rs = 1.00), words spelled correctly ( rs = .99–1.00), correct word sequences (CWS; rs = .96–.97), and correct minus incorrect word sequences (CIWS; rs = .86–.92). For CWS and CIWS, however, automated scores systematically overestimated hand-calculated scores, with an unacceptable amount of error for CIWS for some types of decisions. These findings provide preliminary evidence that aWE-CBM can be used to efficiently score narrative writing samples, potentially improving the feasibility of implementing multi-tiered systems of support in which the written expression skills of large numbers of students are screened and monitored.

scholarly journals Accuracy of Automated Written Expression Curriculum-based Measurement Scoring

2020 ◽  
Author(s):  
Sterett Mercer ◽  
Joanna Cannon ◽  
Bonita Squires ◽  
Yue Guo ◽  
Ella Pinco
Keyword(s):  
Progress Monitoring ◽  
Preliminary Evidence ◽  
Written Expression ◽  
Student Writing ◽  
Curriculum Based Measurement ◽  
Narrative Writing ◽  
Large Numbers ◽  
Community Based Organization ◽  
Highly Correlated ◽  
Total Words Written

We examined the extent to which automated written expression curriculum-based measurement (aWE-CBM) can be accurately used to computer score student writing samples for screening and progress monitoring. Students (n = 174) with learning difficulties in Grades 1–12 who received 1:1 academic tutoring through a community-based organization completed narrative writing samples in the fall and spring across two academic years. The samples were evaluated using four automated and hand-calculated WE-CBM scoring metrics. Results indicated automated and hand-calculated scores were highly correlated at all four timepoints for counts of total words written (rs = 1.00), words spelled correctly (rs = .99 – 1.00), correct word sequences (CWS; rs = .96 – .97), and correct minus incorrect word sequences (CIWS; rs = .86 – .92). For CWS and CIWS, however, automated scores systematically overestimated hand-calculated scores, with an unacceptable amount of error for CIWS for some types of decisions. These findings provide preliminary evidence that aWE-CBM can be used to efficiently score narrative writing samples, potentially improving the feasibility of implementing multi-tiered systems of support in which the written expression skills of large numbers of students are screened and monitored.

scholarly journals Validity of Automated Learning Progress Assessment in English Written Expression for Students with Learning Difficulties

2021 ◽  
Author(s):  
Sterett Mercer ◽  
Joanna Cannon
Keyword(s):  
Progress Monitoring ◽  
Learning Difficulties ◽  
Preliminary Evidence ◽  
Written Expression ◽  
Narrative Writing ◽  
Writing Quality ◽  
Automated Learning ◽  
Learning Progress ◽  
Highly Correlated ◽  
Academic Year

We evaluated the validity of an automated approach to learning progress assessment (aLPA) for English written expression. Participants (n = 105) were students in Grades 2–12 who had parent-identified learning difficulties and received academic tutoring through a community-based organization. Participants completed narrative writing samples in the fall and spring of one academic year, and some participants (n = 33) also completed a standardized writing assessment in the spring of the academic year. The narrative writing samples were evaluated using aLPA, four hand-scored written expression curriculum-based measures (WE-CBM), and ratings of writing quality. Results indicated (a) aLPA and WE-CBM scores were highly correlated with ratings of writing quality; (b) aLPA and more complex WE-CBM scores demonstrated acceptable correlations with the standardized writing subtest assessing spelling and grammar, but not the subtest assessing substantive quality; and (c) aLPA scores showed small, statistically significant improvements from fall to spring. These findings provide preliminary evidence that aLPA can be used to efficiently score narrative writing samples for progress monitoring, with some evidence that the aLPA scores can serve as a general indicator of writing skill. The use of automated scoring in aLPA, with performance comparable to WE-CBM hand scoring, may improve scoring feasibility and increase the likelihood that educators implement aLPA for decision making.

scholarly journals The Potential for Automated Text Evaluation to Improve the Technical Adequacy of Written Expression Curriculum-Based Measurement

2018 ◽  
Vol 42 (2) ◽  
pp. 117-128 ◽  
Author(s):  
Sterett H. Mercer ◽  
Milena A. Keller-Margulis ◽  
Erin L. Faith ◽  
Erin K. Reid ◽  
Sarah Ochs
Keyword(s):  
External Validity ◽  
Progress Monitoring ◽  
Fifth Grade ◽  
Written Expression ◽  
Curriculum Based Measurement ◽  
Structural Validity ◽  
Technical Adequacy ◽  
Fifth Grade Students ◽  
Predictive Algorithms ◽  
Academic Supports

Written-expression curriculum-based measurement (WE-CBM) is used for screening and progress monitoring students with or at risk of learning disabilities (LD) for academic supports; however, WE-CBM has limitations in technical adequacy, construct representation, and scoring feasibility as grade-level increases. The purpose of this study was to examine the structural and external validity of automated text evaluation with Coh-Metrix versus traditional WE-CBM scoring for narrative writing samples (7-min duration) collected in fall and winter from 144 second- through fifth-grade students. Seven algorithms were applied to train models of Coh-Metrix and traditional WE-CBM scores to predict holistic quality of the writing samples as evidence of structural validity; then, external validity was evaluated via correlations with rated quality on other writing samples. Key findings were that (a) structural validity coefficients were higher for Coh-Metrix compared with traditional WE-CBM but similar in the external validity analyses, (b) external validity coefficients were higher than reported in prior WE-CBM studies with holistic or analytic ratings as a criterion measure, and (c) there were few differences in performance across the predictive algorithms. Overall, the results highlight the potential use of automated text evaluation for WE-CBM scoring. Implications for screening and progress monitoring are discussed.

Self-Efficacy Regarding Social Work Competencies

2015 ◽  
Vol 27 (5) ◽  
pp. 594-606 ◽  
Author(s):  
Gary Holden ◽  
Kathleen Barker ◽  
Sofie Kuppens ◽  
Gary Rosenberg
Keyword(s):  
Social Work ◽  
Educational Outcomes ◽  
Self Efficacy ◽  
Empirical Work ◽  
Preliminary Evidence ◽  
Efficient Manner ◽  
Replication Studies ◽  
Large Numbers ◽  
Social Work Programs ◽  
Work Education

Purpose: The need for psychometrically sound measurement approaches to social work educational outcomes assessment is increasing. Method: The research reported here describes an original and two replication studies of a new scale ( N = 550) designed to assess an individual’s self-efficacy regarding social work competencies specified by the Council on Social Work Education as part of the accreditation of social work programs. Results: This new measure, the Self-Efficacy Regarding Social Work Competencies Scale (SERSWCS), generally performed in line with our expectations. Discussion: The SERSWCS is a measure that is based on substantial theoretical and empirical work, has preliminary evidence regarding the psychometric properties of the data it produces, can be used with large numbers of students in an efficient manner, is neither expensive or subject to user restrictions, and provides views of outcomes that have utility for pedagogical considerations at multiple curricular levels.

scholarly journals On the Number of Close-to-Optimal Feature Sets

2006 ◽  
Vol 2 ◽  
pp. 117693510600200 ◽  
Author(s):  
Edward R. Dougherty ◽  
Marcel Brun
Keyword(s):  
Discriminant Analysis ◽  
Microarray Data ◽  
Large Collection ◽  
Natural Manner ◽  
Linear Discriminant ◽  
Feature Sets ◽  
Large Numbers ◽  
Sample Data ◽  
Highly Correlated ◽  
Optimal Feature

The issue of wide feature-set variability has recently been raised in the context of expression-based classification using microarray data. This paper addresses this concern by demonstrating the natural manner in which many feature sets of a certain size chosen from a large collection of potential features can be so close to being optimal that they are statistically indistinguishable. Feature-set optimality is inherently related to sample size because it only arises on account of the tendency for diminished classifier accuracy as the number of features grows too large for satisfactory design from the sample data. The paper considers optimal feature sets in the framework of a model in which the features are grouped in such a way that intra-group correlation is substantial whereas inter-group correlation is minimal, the intent being to model the situation in which there are groups of highly correlated co-regulated genes and there is little correlation between the co-regulated groups. This is accomplished by using a block model for the covariance matrix that reflects these conditions. Focusing on linear discriminant analysis, we demonstrate how these assumptions can lead to very large numbers of close-to-optimal feature sets.

Curriculum-Based Measurement of Reading: An Evaluation of Frequentist and Bayesian Methods to Model Progress Monitoring Data

2017 ◽  
Vol 36 (1) ◽  
pp. 55-73 ◽  
Author(s):  
Theodore J. Christ ◽  
Christopher David Desjardins
Keyword(s):  
Bayesian Methods ◽  
Oral Reading ◽  
Progress Monitoring ◽  
Ordinary Least Squares ◽  
Curriculum Based Measurement ◽  
Least Squares Regression ◽  
Validity And Reliability ◽  
Educational Decisions ◽  
Bayesian Estimates ◽  
Published Research

Curriculum-Based Measurement of Oral Reading (CBM-R) is often used to monitor student progress and guide educational decisions. Ordinary least squares regression (OLSR) is the most widely used method to estimate the slope, or rate of improvement (ROI), even though published research demonstrates OLSR’s lack of validity and reliability, and imprecision of ROI estimates, especially after brief duration of monitoring (6-10 weeks). This study illustrates and examines the use of Bayesian methods to estimate ROI. Conditions included four progress monitoring durations (6, 8, 10, and 30 weeks), two schedules of data collection (weekly, biweekly), and two ROI growth distributions that broadly corresponded with ROIs for general and special education populations. A Bayesian approach with alternate prior distributions for the ROIs is presented and explored. Results demonstrate that Bayesian estimates of ROI were more precise than OLSR with comparable reliabilities, and Bayesian estimates were consistently within the plausible range of ROIs in contrast to OLSR, which often provided unrealistic estimates. Results also showcase the influence the priors had estimated ROIs and the potential dangers of prior distribution misspecification.

Sign in / Sign up

Export Citation Format

Share Document