A Systematic Literature Review of Empiricism and Norms of Reporting in Computing Education Research Literature

Context. Computing Education Research (CER) is critical to help the computing education community and policy makers support the increasing population of students who need to learn computing skills for future careers. For a community to systematically advance knowledge about a topic, the members must be able to understand published work thoroughly enough to perform replications, conduct meta-analyses, and build theories. There is a need to understand whether published research allows the CER community to systematically advance knowledge and build theories. Objectives. The goal of this study is to characterize the reporting of empiricism in Computing Education Research literature by identifying whether publications include content necessary for researchers to perform replications, meta-analyses, and theory building. We answer three research questions related to this goal: (RQ1) What percentage of papers in CER venues have some form of empirical evaluation? (RQ2) Of the papers that have empirical evaluation, what are the characteristics of the empirical evaluation? (RQ3) Of the papers that have empirical evaluation, do they follow norms (both for inclusion and for labeling of information needed for replication, meta-analysis, and, eventually, theory-building) for reporting empirical work? Methods. We conducted a systematic literature review of the 2014 and 2015 proceedings or issues of five CER venues: Technical Symposium on Computer Science Education (SIGCSE TS), International Symposium on Computing Education Research (ICER), Conference on Innovation and Technology in Computer Science Education (ITiCSE), ACM Transactions on Computing Education (TOCE), and Computer Science Education (CSE). We developed and applied the CER Empiricism Assessment Rubric to the 427 papers accepted and published at these venues over 2014 and 2015. Two people evaluated each paper using the Base Rubric for characterizing the paper. An individual person applied the other rubrics to characterize the norms of reporting, as appropriate for the paper type. Any discrepancies or questions were discussed between multiple reviewers to resolve. Results. We found that over 80% of papers accepted across all five venues had some form of empirical evaluation. Quantitative evaluation methods were the most frequently reported. Papers most frequently reported results on interventions around pedagogical techniques, curriculum, community, or tools. There was a split in papers that had some type of comparison between an intervention and some other dataset or baseline. Most papers reported related work, following the expectations for doing so in the SIGCSE and CER community. However, many papers were lacking properly reported research objectives, goals, research questions, or hypotheses; description of participants; study design; data collection; and threats to validity. These results align with prior surveys of the CER literature. Conclusions. CER authors are contributing empirical results to the literature; however, not all norms for reporting are met. We encourage authors to provide clear, labeled details about their work so readers can use the study methodologies and results for replications and meta-analyses. As our community grows, our reporting of CER should mature to help establish computing education theory to support the next generation of computing learners.

Study Behavior in Computing Education—A Systematic Literature Review

As the field of computing education grows and matures, it has become essential to unite computing education and higher education research. Educational research has highlighted that how students study is crucial to their learning progress, and study behaviors have been found to play an important role in students’ academic success. This article presents the main results of a systematic literature review intended to determine what we know about the study behaviors of computing students and the role of educational design in shaping them. A taxonomy of study behaviors was developed and used to clarify and classify the definitions of study behavior, process, strategies, habits, and tactics as well as to identify their relationship to the educational context. The literature search resulted in 107 included papers, which were analyzed according to defined criteria and variables. The review of study behavior terminology found that the same terms are used to describe substantially different study behaviors, and the lack of standard terminology makes it difficult to compare findings from different papers. Furthermore, it was more common for papers to use study behaviors to explain other aspects of students rather than exploring and understanding them. Additionally, the results revealed a tendency to focus on specific educational contexts, predominantly introductory programming courses. Although computing education as a field is well equipped to expand the knowledge about both study behaviors and their connection to the educational context, the lack of common terminology and theories limits the impact. The taxonomy of study behaviors in computing education proposed in this article can contribute to contextualizing the research in such a way that researchers and educators across institutional borders can compare and utilize results. Last, the article outlines some areas for future research and recommendations for practice.

Digital Game-Based Learning and Computational Thinking in P-12 Education

The objective of this chapter is to explore the evolution and opportunities of the emerging field of digital games for programming learning, the challenges and tensions that they present, and how educators may be able to collectively advance this work to benefit student learning. This work summarizes previous empirical evidence concerning the positive impacts and outcomes of digital games in computing education, or even impacts that do not let games to spread. Hence, a systematic literature review is carried out in this context to provide a comprehensive overview of works carried out towards incorporating digital games in order to acquire CT skills or learn basic programming concepts within P12 education. The chapter discusses on the range of indicators and measures used in the 44 selected studies, together with methodological limitations and recommendations for further work in this area.

Digital Games for Computing Education

Digital games are considered an effective and efficient instructional strategy for computing education. However, there are few empirical studies providing sound evidence on the benefits of such educational games. In this respect, the objective of this chapter is to present a comprehensive analysis in order to summarize empirical evidence on the benefits of digital games used as an instructional strategy for computing education. The analysis is based on data collected from 21 case studies that use the MEEGA model, the most commonly used model for educational games' evaluation, evaluating 9 different games, involving a population of 344 students. The results indicate that digital games can yield a positive effect on the learning of computing, providing a pleasant and engaging experience to the students and motivating them to study. These results may guide instructors in the selection of educational games as instructional methods and guide game creators with respect to the development of new games.

Introduction to the Special Section on Justice-Centered Computing Education, Part 1

The ideas we offer below for considering justice-centered computing education point to a broad array of problem-spaces, contexts, and communities that scholars, educators, technologists, and activists might engage with. In exploring and deepening the conversation around this project, the seven articles included in the first volume of this special issue employ diverse theoretical perspectives, methodologies, and frameworks, including but not limited to intersectionality, transformational justice, intercultural computing, ethnocomputing, translanguaging, socially responsible computing, and institutional theory. Across them, rather than consensus on a narrow set of issues, we see the possibilities of a pluralistic and wide-ranging conversation about how we might constitute the meanings of “justice-centered” within computing education, the tools that might be used to produce such meanings, and the actions that might address them.

Intercultural Computing Education: Toward Justice Across Difference

Even in the turn toward justice-oriented pedagogy, computing education tends to overlook the quality of intergroup relationships, which risks entrenching division. In this article, we establish an intercultural approach to computing education, informed by intercultural and peace education, prejudice reduction, and the sociology of racism and ethnicity. We outline three major concerns of intercultural computing: shifting from content toward relationships, from cultural responsiveness to cultural reflexivity, and from identity to identification. For the last, we complicate discourses of race and identity widespread in U.S. education. Drawing from studies of youth programming classes in East Africa and U.S. contexts, we then reflect on our attempts to address the first shift of fostering relationships across difference. We highlight three promising design tactics: intergroup pairing, interdependent programming, and making relational goals explicit. Overall, we find that computing can indeed be a site of intergroup bonding across difference, but that bonding can carry complications and tensions with other equity goals and tactics. Rather than framing justice-oriented CS primarily as changes to the aims of computational learning, we argue that future work should explore making relational goals explicit and teach students how to attend to friction.

Rules of Program Behavior

We propose a framework for identifying, organizing, and communicating learning objectives that involve program semantics. In this framework, detailed learning objectives are written down as rules of program behavior (RPBs). RPBs are teacher-facing statements that describe what needs to be learned about the behavior of a specific sort of programs. Different programming languages, student cohorts, and contexts call for different RPBs. Instructional designers may define progressions of RPB rulesets for different stages of a programming course or curriculum; we identify evaluation criteria for RPBs and discuss tradeoffs in RPB design. As a proof-of-concept example, we present a progression of rulesets designed for teaching beginners how expressions, variables, and functions work in Python. We submit that the RPB framework is valuable to practitioners and researchers as a tool for design and communication. Within computing education research, the framework can inform, among other things, the ongoing exploration of “notional machines” and the design of assessments and visualizations. The theoretical work that we report here lays a foundation for future empirical research that compares the effectiveness of RPB rulesets as well as different methods for teaching a particular ruleset.

Applying a Transformative Justice Approach to Encourage the Participation of Black and Latina Girls in Computing

Global protests and civil unrest in 2020 has renewed the world’s interest in addressing injustice due to structural racism and oppression toward Black and Latinx people in all aspects of society, including computing. In this article, we argue that to address and repair the harm created by institutions, policies, and practices that have systematically excluded Black and Latina girls from computer science, an intersectional, transformative justice approach must be taken. Leveraging testimonial authority, we share our past 8 years of experience designing, implementing, and studying Digital Youth Divas, a programmatic and systemic approach to encouraging middle school Black and Latina girls to participate in STEM. Specifically, we propose three principles to counter structural racism and oppression embedded in society and computing education: computing education must (1) address local histories of injustice by engaging community members; (2) counter negative stereotypes perpetuated in computer science by creating inclusive safe spaces and counter-narratives; and (3) build sustainable, computational capacity in communities. To illustrate each principle, we provide specific examples of the harm created by racist policies and systems and their effect on a specific community. We then describe our attempt to create counter structures and the subsequent outcomes for the girls, their families, and the community. This work contributes a framework for STEM and computing educators to integrate transformative justice as a method of repairing the harm that both society and the field of computing has and continues to cause Black and Latinx communities. We charge policy makers, educators, researchers, and community leaders to examine histories of oppression in their communities and to adopt holistic, transformative approaches that counter structural oppression at the individual and system level.