Automated analysis of cognitive presence in MOOC discussions

The Community of Inquiry (CoI) framework [1] has been broadly used to analyse learning experience in online discussion forums for two decades. Cognitive presence, which is a primary dimension of the CoI framework, manifests the reflection of (re)constructing knowledge and problem-solving processes in the learning experience [2]. Researchers doing text analysis using machine learning techniques are making promising contributions to analysing phases of cognitive presence automatically [3]–[5] in online discussions. However, most studies of automated cognitive analysis focus on improving the accuracy and reliability of the classifiers. They ignored that another purpose of applying machine learning techniques in educational research should be to pinpoint research bias that scholars neither intended to nor can have found without computer support. This session will present the example of ‘research bias’ discovered from both manual and automated classification of cognitive phases, provoking scholars to rethink and improve the conflicting part in the taxonomies of cognitive presence under MOOC context. The manual-classification rubric that used to label discussion messages of a target MOOC combines Garrison, Anderson and Archer’s [2] scheme with Park’s [6] revised version. The rubric describes four phases of cognitive presence (i.e. triggering event, exploration, integration and resolution), and indicators of each phase in online discussions. We reported the average inter-rater reliability between two human raters achieved 95.4% agreement (N = 1002) with a Cohen’s weighted kappa of 0.96. Interestingly, we found the average inter-rater reliability decreased to 80.1% after increasing the size of data samples (N = 1918) and the number of human raters to three. After training the automated classifiers to predict phases of cognitive presence, the confusion matrix implies that most of the disagreements between computer raters occurred between adjacent phases of cognitive presence. The disagreements between human raters also have the same problems. We assume the additional categories may exist between cognitive phases in such MOOC discussion messages. These details will be discussed during the presentation. References [1] D. Garrison, T. Anderson, and W. Archer, “Critical Inquiry in a Text-Based Environment: Computer Conferencing in Higher Education,” Internet High. Educ., vol. 2, no. 2, pp. 87–105, 1999. [2] D. Garrison, T. Anderson, and W. Archer, “Critical thinking, cognitive presence, and computer conferencing in distance education,” Am. J. Distance Educ., vol. 15, no. 1, pp. 7–23, 2001. [3] V. Kovanović, S. Joksimović, D. Gašević, and M. Hatala, “Automated cognitive presence detection in online discussion transcripts,” in Automated cognitive presence detection in online discussion transcripts’ CEUR Workshop Proceedings (vol. 1137), 2014. [4] V. Kovanović et al., “Towards automated content analysis of discussion transcripts,” Proc. Sixth Int. Conf. Learn. Anal. Knowl. - LAK ’16, pp. 15–24, 2016. [5] E. Farrow, J. Moore, and D. Gasevic, “Analysing discussion forum data: a replication study avoiding data contamination,” 9th Int. Learn. Anal. Knowl. Conf., no. March, 2019. [6] C. Park, “Replicating the Use of a Cognitive Presence Measurement Tool,” J. Interact. Online Learn., vol. 8, no. 2, pp. 140–155, 2009.

Positive Impact of Computer Conferencing in Distance Education

A number of studies have examined the dynamics and style of communication adopted in computer conferencing in distance education (Mason, 1993). Computer conferencing by higher education institutions offering distance education courses has expanded rapidly since 1987. Lately computer conferencing has developed in fame as a vehicle of distance education. While protecting the opportunity related with asynchronous communication and giving wide access to low-end innovation users, computer conferencing enables students and instructors to take part in group learning and associate 'many-to-many' (Harasim, 1989). Much has already been written on the subject of computer conferencing in support of educational processes, in particular distance education (DE). Using unstructured interviews and observations at adult students' homes or worksites, the study investigated adult student perspectives of distance study by computer conferencing. The focal finding of this investigation was that computer conferencing can bolster a scope of learning situations, from teacher-led symposium to symposium to a student-centered community workshop. In addition, adults actively engaged in social relationships outside their distance studies which sustain their educational pursuits. The students provided insights into aspects of the on‐line environment: as synchronicity, interactivity, textual communications, and collaboration.

VIRTUAL WRITING FORUM WITH DON MURRAY AND THE NATIONAL WRITING PROJECT IN AN ASYNCHRONOUS ENVIRONMENT

This study evaluated the level of participation and perceptions of effectiveness and value among participants in a virtual forum interacting with one another and with Donald Murray, a noted mentor in writing instruction. National Writing Project teacher consultants were invited to participate in a two-week interactive session using Caucus®, a non-threaded, World Wide Web-based, asynchronous computer conferencing system. Eighty-one teacher-consultants responded and participated in the event. Using a multiple perspective framework, data gathered through surveys, interviews, transcript analysis and online discussions suggest that participants, project coordinators and directors, as well as the author himself, found the event a valuable learning experience with interesting possibilities.

ASSESSING TEACHING PRESENCE IN A COMPUTER CONFERENCING CONTEXT

This paper presents a tool developed for the purpose of assessing teaching presence in online courses that make use of computer conferencing, and preliminary results from the use of this tool. The method of analysis is based on Garrison, Anderson, and Archer’s model of critical thinking and practical inquiry in a computer conferencing context. The concept of teaching presence is constitutively defined as having three categories – design and organization, facilitating discourse, and direct instruction. Indicators that we search for in the computer conference transcripts identify each category. Pilot testing of the instrument reveals interesting differences in the extent and type of teaching presence found in different graduate level online courses.