Evaluating Content-Management Systems for Online Learning Programs

Creating quality assessments typically requires the involvement of many people who require access to the item and test information, which is stored in repositories called item banks or, more appropriately, content-management systems, since they store many kinds of content used in the test development process. This chapter discusses the types of options that are available in content-management systems and provides guidance about how to evaluate whether different content-management systems will meet an organization’s test development and delivery needs. This chapter focuses on online, fully Internet-enabled applications, since those applications have the most features.

Testing the Validity of the Post and Vote Model of Web-Based Peer Assessment

Two tests of validity were conducted with undergraduate education students on a method of online peer assessment called post and vote. Validity was determined by calculating a Pearson product-moment correlation and corresponding coefficient of determination that compared the average grade assigned by the pre-service teachers with the grade assigned independently by the course instructor. Results of both studies showed that post and vote Web-based peer assessment was valid with these groups, and generalizable to undergraduate classes engaged in similar tasks.

Performance Testing

Performance testing evaluates real-world tasks and skills with a test-display and response environment similar to, or identical with, the job environment. Performance testing offers the promise of providing more comprehensive evidence of construct and predictive validity than knowledge-based testing. This chapter presents information relevant to this promising approach to online testing. First, performance testing is defined and a historical context is presented; a series of test design questions are presented, then validity criteria, standards, and theory are recommended; and two validity-centered design approaches are reviewed. The chapter concludes with a set of implications and conclusions for further investigation.

Using Messicks Framework to Validate Assessment Tasks in Online Environments

Messick (1988) maintained that technology-based delivery methods would transform both our conceptions of teaching and learning, and also our methods for evaluating student learning. Designed 100 years ago (Crocker & Algina, 1986), classical approaches to assessing test quality are unsuited to the contemporary context of technology-based learning (Messick, 1988). We will discuss evolving conceptions of validity and show how Messick’s (1989) framework is an improvement over traditional conceptions. We will then apply Messick’s framework to the evaluation data from the globally delivered, hybrid “A Course in Writing Effectively for UNHCR.” Our results will show how Messick’s framework provides a comprehensive assessment, based on evidence, values, and consequences of the merit and worth of contemporary assessment tasks.

Variations in Adaptive Testing and Their Online Leverage Points

This chapter builds on foundational work on probabilistic frames of reference and principled assessment design to explore the role of adaptation in assessment. Assessments are characterized in terms of their claim status, observation status, and locus of control. The relevant claims and observations constitute a frame of discernment for the assessment. Adaptation occurs when the frame is permitted to evolve with respect to the claims or observations (or both); adaptive features may be controlled by the examiner or the examinee. In describing the various combinations of these characteristics, it is argued that an online format is preeminent for supporting common and emerging assessment practices in light of adaptation.

The Role of Assessment and Evaluation in Context

The purpose of this chapter is to facilitate discussion regarding the contextualization of assessment and evaluation within online educational environments. Three case studies will be used to frame questions, issues, and challenges regarding online assessment and evaluation that were influenced by pedagogical alignment and environmental affordances and constraints. These reflective case studies, highlighting the author’s experiences as a teacher, researcher, and instructional design team leader, will be used to illustrate assessment and evaluation within three very different educational contexts: an online K-12 public school, an undergraduate online course, and a learning community designing online graduate courses.

Authentic Assessment Online

This chapter addresses the issue of authentic assessment in an online, asynchronous educational environment. Initially, a definition of “authentic” is articulated. Building on this theoretical foundation by describing authentic assessment in the face-to-face classroom, a framework for authentic assessment in the virtual classroom is developed. Next, the multiple challenges of conducting authentic assessment online are addressed. Finally, specific strategies for authentic online assessment are identified and discussed. These strategies include the use of electronic portfolios, electronic journals, assessment embedded in online discussion, and rubrics.

Online Assessment in a Teacher Education Program

In this chapter the authors describe how they have incorporated a constructivist view of learning into their approach to online assessment. The context of their discussion is within an accelerated (12-month) teacher education program at the graduate level for preservice secondary teachers. At the heart of the authors’ view of assessment is an attempt to capture through the use of online techniques how their preservice teachers are progressing toward becoming effective teachers through engagement in authentic situations that are experience based and problem oriented. The authors present three examples of how they use online assessment to encourage elaborated discussion, metacognitive thinking, and critical reflection.

Online program Assessment

Through case study, this chapter lends insight to ways online assessment can facilitate a holistic approach to the evaluation of distance education programs. In 2001, the University of Illinois at Urbana-Champaign transitioned from program evaluation methods that relied heavily on data gathering by postal mail to online instruments. While the transition was spurred by the need to evaluate the campus’ first online degree program, online assessment methods are now used to review all off-campus degree programs. Results of this new assessment strategy have proven beneficial for continuous quality improvement across all modes of delivery.

Online Course-Ratings and the Personnel Evaluation Standards

Many institutions of higher education are considering the possibility of conducting student evaluations of teaching (course-ratings) online. Some campuses have already implemented online evaluation systems that collect, process, and report ratings data electronically. Information on the successes and challenges of these systems is beginning to emerge. This chapter outlines some of the most salient advantages and challenges of online student evaluations of teaching within the context of how they relate to The Personnel Evaluation Standards set forth by the Joint Committee on Standards for Educational Evaluation (JCSEE, 1988). The authors also provide suggestions for successful implementation of online evaluation systems.