scholarly journals Methodological Concerns About the Education Value-Added Assessment System (EVAAS): Validity, Reliability, and Bias

SAGE Open ◽  
2020 ◽  
Vol 10 (2) ◽  
pp. 215824402092222
Author(s):  
Audrey Amrein-Beardsley ◽  
Tray Geiger
Keyword(s):  
Value Added ◽  
Assessment System ◽  
School Level ◽  
Business Analytics ◽  
Validity And Reliability ◽  
Independent School District ◽  
Value Added Models ◽  
Education Value ◽  
Composition Factors ◽  
Student Composition

The Education Value-Added Assessment System (EVAAS), the value-added model (VAM) sold by the international business analytics software company SAS Institute Inc., is advertised as offering “precise, reliable and unbiased results that go far beyond what other simplistic [value-added] models found in the market today can provide.” In this study, we investigated these claims, as well as others pertaining to the validity or truthfulness of model output, by conducting analyses on more than 1,700 teachers’ EVAAS results (i.e., actual EVAAS output to which no other external scholars have had access prior) from the Houston Independent School District (HISD). We found the EVAAS to perform, overall, in line with other VAMs in terms of validity and reliability, although it yielded possibly more biased value-added estimates than other VAMs due to differences in teacher’s EVAAS scores based on school-level student composition factors.

Methodological Concerns About the Education Value-Added Assessment System

2008 ◽  
Vol 37 (2) ◽  
pp. 65-75 ◽  
Author(s):  
Audrey Amrein-Beardsley
Keyword(s):  
Value Added ◽  
Assessment System ◽  
Regression To The Mean ◽  
Methodological Issues ◽  
User Friendliness ◽  
Paradigm Case ◽  
Student Background ◽  
Data Regression ◽  
Value Added Models ◽  
Education Value

Value-added models help to evaluate the knowledge that school districts, schools, and teachers add to student learning as students progress through school. In this article, the well-known Education Value-Added Assessment System (EVAAS) is examined. The author presents a practical investigation of the methodological issues associated with the model. Specifically, she argues that, although EVAAS is probably the most sophisticated value-added model, it has flaws that must be addressed before widespread adoption. She explores in depth the shortage of external reviews and validity studies of the model, its insufficient user-friendliness, and methodological issues about missing data, regression to the mean, and student background variables. She also examines a paradigm case in which the model was used to advance unfounded assertions.

scholarly journals The SAS Education Value-Added Assessment System (SAS® EVAAS®) in the Houston Independent School District (HISD): Intended and Unintended Consequences

2012 ◽  
Vol 20 ◽  
pp. 12 ◽  
Author(s):  
Audrey Amrein-Beardsley ◽  
Clarin Collins
Keyword(s):  
School District ◽  
Independent School ◽  
Value Added ◽  
Unintended Consequences ◽  
Assessment System ◽  
Unintended Effects ◽  
High Stakes ◽  
Educational Value ◽  
Independent School District ◽  
Houston Independent School District

The SAS Educational Value-Added Assessment System (SAS® EVAAS®) is the most widely used value-added system in the country. It is also self-proclaimed as “the most robust and reliable” system available, with its greatest benefit to help educators improve their teaching practices. This study critically examined the effects of SAS® EVAAS® as experienced by teachers, in one of the largest, high-needs urban school districts in the nation – the Houston Independent School District (HISD). Using a multiple methods approach, this study critically analyzed retrospective quantitative and qualitative data to better comprehend and understand the evidence collected from four teachers whose contracts were not renewed in the summer of 2011, in part given their low SAS® EVAAS® scores. This study also suggests some intended and unintended effects that seem to be occurring as a result of SAS® EVAAS® implementation in HISD. In addition to issues with reliability, bias, teacher attribution, and validity, high-stakes use of SAS® EVAAS® in this district seems to be exacerbating unintended effects.

“Houston, We Have a Lawsuit”: A Cautionary Tale for the Implementation of Value-Added Models for High-Stakes Employment Decisions

2020 ◽  
Vol 49 (5) ◽  
pp. 350-359 ◽  
Author(s):  
Mark A. Paige ◽  
Audrey Amrein-Beardsley
Keyword(s):  
Due Process ◽  
Value Added ◽  
Poor Performance ◽  
Federal Court ◽  
High Stakes ◽  
Employment Decisions ◽  
Independent School District ◽  
Procedural Due Process ◽  
Value Added Models ◽  
Federal Constitutional

Until recently, legal challenges to the use of value-added models (VAMs) in evaluation and teacher employment decisions in federal court had been unsuccessful. However, in May 2017 a federal court in Texas ruled that plaintiff-teachers established a viable federal constitutional claim to challenge the use of VAMs as a means for their termination in Houston Federation of Teachers v. Houston Independent School District. Houston represents a significant departure from prior federal court rulings that upheld the constitutionality of VAMs to terminate teachers on the basis of poor performance. The Houston court found that the districts’ refusals to release the underlying data of VAM ratings used to terminate those teachers violated the teachers’ procedural due process rights. By denying access to the code, teachers could not protect against the government’s making a mistaken deprivation of their property right to continued right to employment. The authors discuss Houston and its potential impact, limitations, and significance.

scholarly journals Using the TxVAAS to Improve Teacher Effectiveness: Investigating the Research-Situated “Truths” Behind TxVAAS Claims

2018 ◽  
Vol 8 (1) ◽  
Author(s):  
Audrey Amrein-Beardsley ◽  
Clarin Collins
Keyword(s):  
Content Analysis ◽  
Teacher Effectiveness ◽  
Value Added ◽  
Psychological Testing ◽  
Assessment System ◽  
Education Value

At the time of this study, the Texas Value-Added Assessment System (TxVAAS) was being piloted throughout Texas to hold teachers more accountable for their effects on students' achievement (i.e., teachers’ value added). It is still being used by districts throughout Texas today. Using a framework informed by the<em> Standards for Educational and Psychological Testing</em>, researchers conducted a content analysis of the marketing and research-based claims asserted about the TxVAAS, to (a) examine the “truth” of each claim to (b) help others critically consume the marketing claims using (c) nonproprietary, peer-reviewed literature. Given that the more popular, and also proprietary version of the TxVAAS—the Education Value-Added Assessment System—continues to be sold and marketed to other states and districts in similar ways, researchers deemed it critical to intervene before other states and districts might blindly trust the marketing and research-based claims presented.

All sizzle and no steak

2017 ◽  
Vol 99 (2) ◽  
pp. 53-59 ◽  
Author(s):  
Audrey Amrein-Beardsley ◽  
Tray Geiger
Keyword(s):  
English Language Learners ◽  
Language Learners ◽  
English Language ◽  
Value Added ◽  
Special Education Students ◽  
Assessment System ◽  
Education Students ◽  
High Stakes ◽  
Educational Value ◽  
Value Added Models

Houston’s experience with the Educational Value-Added Assessment System (R) (EVAAS) raises questions that other districts should consider before buying the software and using it for high-stakes decisions. Researchers found that teachers in Houston, all of whom were under the EVAAS gun, but who taught relatively more racial minority students, higher proportions of English language learners, higher proportions of students eligible for free or reduced-price lunches, and higher proportions of special education students, had significantly lower EVAAS scores than colleagues teaching elsewhere in the Houston district. Hence, results suggest that the EVAAS does not, at least in Houston and perhaps elsewhere, offer states, districts, and schools the precise, reliable, and unbiased results that go far beyond what other simplistic [value-added] models found in the market today can provide, as the software owner, SAS Institute Inc., claims. Rather, evidence shows that EVAAS estimates in Houston, and likely elsewhere, may be biased against teachers who teach disproportionate percentages of certain type of students in their classrooms.

The Impact of Summer Learning Loss on Measures of School Performance

2017 ◽  
Vol 12 (4) ◽  
pp. 468-491 ◽  
Author(s):  
Andrew McEachin ◽  
Allison Atteberry
Keyword(s):  
School Quality ◽  
Value Added ◽  
School Level ◽  
Summer Learning Loss ◽  
Learning Loss ◽  
Education Policies ◽  
Summer Learning ◽  
School Year ◽  
Value Added Models ◽  
The Impact

State and federal accountability policies are predicated on the ability to estimate valid and reliable measures of school impacts on student learning. The typical spring-to-spring testing window potentially conflates the amount of learning that occurs during the school year with learning that occurs during the summer. We use a unique dataset to explore the potential for students’ summer learning to bias school-level value-added models used in accountability policies and research on school quality. The results of this paper raise important questions about the design of performance-based education policies, as well as schools’ role in the production of students’ achievement.

Sign in / Sign up

Export Citation Format

Share Document