Assessing the Quality of Education Research Through Its Relevance to Practice: An Integrative Review of Research-Practice Partnerships

The contemporary social, economic, and cultural conditions within and outside the academy prompt important questions about the role of research in education policy and practice. Scholars have framed research-practice partnerships (RPPs) as a strategy to promote evidence-based decision-making in education. In this chapter, I interrogate the notion that RPPs offer an insightful framework to consider how the quality of research can be measured through its use. The findings suggest that using RPPs to assess the quality of education research enhances the relevance to policy and practice as well as attention to the quality of reporting, and pivots from the preeminence of methodological quality. RPPs increase local education leaders’ access to research and bolster the use of research. RPPs may also strengthen the alignment between education research and the public good. Notwithstanding, employing RPPs as a vehicle to assess research quality has its challenges. Valuing the work of RPPs in academia is a work in progress. Building and sustaining an RPP is challenging, and there is still much to learn about the ways in which RPPs work and overcome obstacles. Assessing the impact of RPPs is also difficult. Future considerations are discussed.

Australian Research in Early Childhood Education and Care: Insights Into the Actual; Imagining the Possible

Early childhood education and care (ECEC) remains a priority area for public policy, internationally and in Australia. However, an analysis of empirical research published internationally up to 2008 has identified a bias toward positivist methodologies within a “scientific/psychological’ rather than educational perspective and with a focus on the interactions between preschoolers, family, and child care variables. For some researchers, this bias raises concerns that public policy in ECEC is based on limited research perspectives. This chapter examines research focusing on the Australian context and published between 2010 and 2014 to determine whether this bias exists in Australian research. We explore the quality of ECEC research to develop an overall understanding of the current situation of ECEC research in Australia. Our findings suggest that Australian research in ECEC is very dissimilar to research published internationally, especially in its reliance on qualitative paradigms and a focus on the educators (principals, teachers, and teacher aides). The strong qualitative focus may allow a diverse range of voices within the ECEC sector to be heard and identified, moving beyond traditional notions of historically marginalized individuals and communities that dominate other education research areas.

Considerations for Evidence Frameworks in Education Research

In this chapter, we describe and compare the standards for evidence used by three entities that review studies of education interventions: Blueprints for Healthy Youth Development, Social Programs that Work, and the What Works Clearinghouse. Based on direct comparisons of the evidence frameworks, we identify key differences in the level at which effectiveness ratings are granted (i.e., intervention vs. outcome domain), as well as in how each entity prioritizes intervention documentation, researcher independence, and sustained versus immediate effects. Because such differences in priorities may result in contradictory intervention ratings between entities, we offer a number of recommendations for a common set of standards that would harmonize effectiveness ratings across the three entities while preserving differences that allow for variation in user priorities. These include disentangling study rigor from intervention effectiveness, ceasing vote counting procedures, adding replication criteria, adding fidelity criteria, assessing baseline equivalence for randomized studies, making quasi-experiments eligible for review, adding criteria for researcher independence, and providing effectiveness ratings at the level of the outcome domain rather than the intervention.

Meeting the Needs of All Cultureless Learners: Culture Discourse and Quality Assumptions in Personalized Learning Research

This chapter reviews recent qualitative studies on personalized learning in middle/secondary school settings to analyze the role of culture in how this concept is enacted and researched. Personalized learning is posited as a pedagogical approach that aims to revolutionize schooling and challenge educational inequity by foregrounding learners’ agency in what and how they learn, tailoring pedagogy and its purpose to learners’ unique interests, needs, and abilities. Given the strong emphasis of the approach on the uniquenesses of the persons who are learning, our analysis interrogates the discourse on culture in studies on personalized learning and extrapolates how this discourse informs problem formulation, design and logic, sources of evidence, analysis and interpretation, and implications for practice. This review reveals a disconnect between the relevant literature on culture in learning and omissions of researchers and research participants’ cultural positionalities and identities. This appears to affect the quality of educational evidence, inhibiting a deep understanding of the implementation of the personalized learning approach for different communities of learners. We assert that research into practices that intend to meet the needs of diverse learners should center learner and researcher cultures and positionalities as part of a theory of change that permeates the entire research process.

Toward a Science of Failure Analysis: A Narrative Review

When a researcher tests an educational program, product, or policy in a randomized controlled trial and detects a significant effect on an outcome, the intervention is usually classified as something that “works.” When expected effects are not found, there is seldom an orderly and transparent analysis of plausible reasons why. Accumulating and learning from possible failure mechanisms is not standard practice in education research, and it is not common to design interventions with causes of failure in mind. This chapter develops Boruch and Ruby’s proposition that the education sciences would benefit from a systematic approach to the study of failure. We review and taxonomize recent reports of large-scale randomized controlled trials in K–12 schooling that yielded at least one null or negative major outcome, including the nature of the event and reasons (if provided) for why it occurred. Our purpose is to introduce a broad framework for thinking about educational interventions that do not produce expected effects and seed a cumulative knowledge base on when, how, and why interventions do not reach expectations. The reasons why an individual intervention fails to elicit an outcome are not straightforward, but themes emerge when researchers’ reports are synthesized.

Evidence-Based Practices in Deaf Education: A Call to Center Research and Evaluation on the Experiences of Deaf People

The evidence base for educational interventions for deaf students has been, and continues to be, called into question due to a lack of “gold standard” research available to support it. Yet the paucity of research in deaf education is not only in the volume of research that meets rigorous standards but also in its lack of attention to and inclusion of a deaf-centered perspective on the inferences made about the strength of study findings in the field. This chapter uses a deaf-centered lens to examine what constitutes evidence, how it is gained, and how this information supports academic outcomes for this population. We include examples from the literature to examine implications for research personnel, study design, and accessible dissemination, with specific attention to both study sampling and measurement considerations. Considerations for deaf-centered research criteria include (a) integrating deaf researchers and epistemologies, (b) attending to the characteristics of deaf students, and (c) acknowledgment of root causes and systems factors. The recommendations in this chapter supplement the larger ongoing dialogue regarding the cultural responsiveness and representation of marginalized populations within the education research endeavor.

Research Worth Using: (Re)Framing Research Evidence Quality for Educational Policymaking and Practice

This chapter calls for researchers to reconceptualize research quality from the perspective of its expected use, attending to power dynamics that influence how knowledge is defined, constructed, and validated through the research enterprise. Addressing these concerns when designing and conducting education research can yield more useful research evidence for building more equitable education systems. Anchored in scholarship on research utilization and methodological critiques, the chapter introduces a research quality framework that integrates relevance and rigor through five key dimensions of Research Worth Using: (1) relevance of question: alignment of research topics to practical priorities; (2) theoretical credibility: explanatory strength and coherence of principles investigated; (3) methodological credibility: internal and external credibility of study design and execution; (4) evidentiary credibility: robustness and consistency of cumulative evidence; and (5) relevance of answers: justification for practical application. This framework simultaneously uplifts the voices and needs of policymakers, practitioners, and community members, while elevating standards for excellence in education research. We call attention to the myriad ways in which the quality of evidence generated can be strengthened, before describing implications for curating and using research. We conclude by offering suggestions for applying and further developing the framework.

Understanding Disability: High-Quality Evidence in Research on Special Education Disproportionality

This chapter examines how studies focused on the same topic—disproportionality in special education—can generate vastly different conclusions about its sources and causes. By analyzing existing disagreements in the field, we explore essential questions about what constitutes high-quality and relevant evidence when seeking to understand how, when, for whom, and why disproportionality occurs. Using a holistic review of the empirical literature on disproportionality, we illustrate how differing epistemological and ontological views inform research around the topic of disability in schools and argue that to develop high-quality evidence around disproportionality, researchers need a shared framework that describes how school-based disabilities and classification processes intersect. A shared framework will enable researchers to evaluate whether their findings are expected or unexpected, connect to other related research, and build and rebuild paradigms around issues of equity in special education, rather than disregard one set of findings over another.

The Wretched of the Research: Disenchanting Man2-as-Educational Researcher and Entering the 36th Chamber of Education Research

Compulsory state-sanctioned schooling continues to be constructed as the “great equalizer,” and accordingly education research as a benevolent contributor to this material and ideological project of education. Following a Fanonian-Wynterian theoretical approach and cosmogonical-constellatory citation politics, I narrowed over 2,500 educational studies and reviewed approximately 150 articles and chapters that questioned the ways of knowing, being, and valuing which have naturalized these assumptions. Consequently, I theorize the cosmogony and development of the overrepresented genre-specific figure of educational researcher emerging from Man2-as-human, who has come to control the ways of knowing “education” and being an “educational researcher”: Man2-as-educational researcher. I examine how overlapping and interconnected African/Black, Asian, Latinx, Pacific Islander and Indigenous communities have engaged in modes of resistance, survivance, fugitivity/marronage, refusal and abolition to challenge this regime, and enact and imagine genres of being an educational researcher outside of the dominant order of Man2-as-educational researcher. In turn, I consider how these communities have affirmed, honored, fostered, sustained and revitalized ways of gathering, interpreting, and sharing educational knowledge for collective liberation, which have centered the wretched of the research and gaze from below. In so doing, I conceptualize and call forth the need to move toward what I am referring to as the 36th chamber of education research.