Examining the Validity of Adaptive Comparative Judgment for Peer Evaluation in a Design Thinking Course

Adaptive comparative judgment (ACJ) is a holistic judgment approach used to evaluate the quality of something (e.g., student work) in which individuals are presented with pairs of work and select the better item from each pair. This approach has demonstrated high levels of reliability with less bias than other approaches, hence providing accurate values in summative and formative assessment in educational settings. Though ACJ itself has demonstrated significantly high reliability levels, relatively few studies have investigated the validity of peer-evaluated ACJ in the context of design thinking. This study explored peer-evaluation, facilitated through ACJ, in terms of construct validity and criterion validity (concurrent validity and predictive validity) in the context of a design thinking course. Using ACJ, undergraduate students (n = 597) who took a design thinking course during Spring 2019 were invited to evaluate design point-of-view (POV) statements written by their peers. As a result of this ACJ exercise, each POV statement attained a specific parameter value, which reflects the quality of POV statements. In order to examine the construct validity, researchers conducted a content analysis, comparing the contents of the 10 POV statements with highest scores (parameter values) and the 10 POV statements with the lowest scores (parameter values)—as derived from the ACJ session. For the criterion validity, we studied the relationship between peer-evaluated ACJ and grader’s rubric-based grading. To study the concurrent validity, we investigated the correlation between peer-evaluated ACJ parameter values and grades assigned by course instructors for the same POV writing task. Then, predictive validity was studied by exploring if peer-evaluated ACJ of POV statements were predictive of students’ grades on the final project. Results showed that the contents of the statements with the highest parameter values were of better quality compared to the statements with the lowest parameter values. Therefore, peer-evaluated ACJ showed construct validity. Also, though peer-evaluated ACJ did not show concurrent validity, it did show moderate predictive validity.

Attention alters certainty but not appearance

Does paying attention to a stimulus change its appearance or merely influence the decision mechanisms involved in reporting it? Recently we proposed an uncertainty stealing hypothesis in which subjects, when uncertain about a perceptual comparison between a cued and uncued stimulus, tend to disproportionately choose the cued stimulus. The result is a psychometric function that mimics the results that would be measured if attention actually changed the appearance of the cued stimulus. In the present study, we measure uncertainty explicitly. In three separate experiments, subjects judged the relative appearance of two Gabor patches that differed in contrast. In the first two experiments, subjects performed a comparative judgment, reporting which stimulus had the higher contrast. In the third experiment, subjects performed an equality judgment, reporting whether the two stimuli had the same or different contrast. In the first comparative judgment experiment and in the equality judgment experiment, one of the two stimuli was pre-cued by an exogenous cue. In the second comparative judgment experiment, a decision bias was explicitly introduced: one stimulus was followed by a post-cue and the subjects were instructed, when uncertain, to choose the cued target. In all three experiments, subjects also indicated whether or not they were certain about each response. The results reveal that in the pre-cue comparative judgment, attention shifted the subjects’ uncertainty and made subjects more likely to report that the cued stimulus had higher contrast. In the post-cue biased comparative judgment, subjects also were more likely to report that the cued stimulus had higher contrast, but without a shift in uncertainty. In the equality judgment, attention did not affect the contrast judgment, and the subjects’ uncertainty remained aligned with their decision. We conclude that attention does not alter appearance but rather manipulates subjects’ uncertainty and decision mechanisms.

On the bias and stability of the results of comparative judgment

Comparative judgment is a method that allows measurement of a competence by comparison of items with other items. In educational measurement, where comparative judgment is becoming an increasingly popular assessment method, items are mostly students’ responses to an assignment or an examination. For assessments using comparative judgment, the Scale Separation Reliability (SSR) is used to estimate the reliability of the measurement. Previous research has shown that the SSR may overestimate reliability when the pairs to be compared are selected with certain adaptive algorithms, when raters use different underlying models/truths, or when the true variance of the item parameters is below one. This research investigated bias and stability of the components of the SSR in relation to the number of comparisons per item to increase understanding of the SSR. We showed that many comparisons are required to obtain an accurate estimate of the item variance, but that the SSR can be useful even when the variance of the items is overestimated. Lastly, we recommend adjusting the general guideline for the required number of comparisons per item to 41 comparisons per item. This recommendation partly depends on the number of items and the true variance in our simulation study and needs further investigation.

Effective uncertainty visualization for aftershock forecast maps

Earthquake models can produce aftershock forecasts, which have recently been released to lay audiences following large earthquakes. While visualization literature suggests that displaying forecast uncertainty can improve how forecast maps are used, research on uncertainty visualization is missing from earthquake science. We designed a pre-registered online experiment to test the effectiveness of three visualization techniques for displaying aftershock forecast maps and their uncertainty. These maps showed the forecasted number of aftershocks at each location for a week following a hypothetical mainshock, along with the uncertainty around each location’s forecast. Three different uncertainty visualizations were produced: (1) forecast and uncertainty maps adjacent to one another; (2) the forecast map depicted in a color scheme, with the uncertainty shown by the transparency of the color; and (3) two maps that showed the lower and upper bounds of the forecast distribution at each location. Unlike previous experiments, we compared the three uncertainty visualizations using tasks that are systematically designed to address broadly applicable and user-generated communication goals. We compared task responses between participants using uncertainty visualizations and using the forecast map shown without its uncertainty (the current practice). Participants completed two map-reading tasks that targeted several dimensions of the readability of uncertainty visualizations. Participants then performed a comparative judgment task, which demonstrated whether a visualization was successful in reaching two key communication goals: indicating where many aftershocks and no aftershocks are likely (sure bets) and where the forecast is low but the uncertainty is high enough to imply potential risk (surprises). All visualizations performed equally well in the goal of communicating sure bet situations. But the visualization with lower and upper bounds was substantially better than the other designs at communicating surprises. These results have implications for the communication of forecast uncertainty both within and beyond earthquake science.

A comparative judgment approach to assessing Chinese Sign Language interpreting

The quality of sign language interpreting (SLI) is a gripping construct among practitioners, educators and researchers, calling for reliable and valid assessment. There has been a diverse array of methods in the extant literature to measure SLI quality, ranging from traditional error analysis to recent rubric scoring. In this study, we want to expand the terrain of SLI assessment, by exploring and evaluating a novel method, known as comparative judgment (CJ), to assess SLI quality. Briefly, CJ involves judges to compare two like objects/items and make a decision by choosing the one with higher quality. The binary outcomes from repeated comparisons by a group of judges are then modelled statistically to produce standardized estimates of perceived quality for each object/item. We recruited 12 expert judges to operationalize CJ via a computerized system to assess the quality of Chinese Sign Language interpreting produced by 36 trainee interpreters. Overall, our analysis of quantitative and qualitative data provided preliminary evidential support for the validity and utility of CJ in SLI assessment. We discussed these results in relation to previous SLI literature, and suggested future research to cast light on CJ’s usefulness in applied assessment contexts.

A Visual Design Framework and Assessment to Inform Instruction

As evidence of meeting program criteria required by the Accreditation Board for Engineering and Technology (ABET), engineering technology (ET) students are expected to both read and create graphical communications. Academic librarians’ visual literacy (VL) instruction can support ET students’ ability to communicate through graphics. Under comprehensive VL instruction, teaching visual design principles is an area of VL that supports graphical communication. An adaptive comparative judgment (ACJ) assessment was given to 115 ET students to inform visual design instruction and future research. A visual design framework supplemented the assessment to familiarize students with visual design principles. ACJ offers an alternative assessment model because instead of grading against a rubric of learning outcomes, it uses context to judge the quality of a work. The assessment results outline specific areas to focus visual design instruction for students to effectively navigate and create graphical communications.

Examining the reliability of Adaptive Comparative Judgement (ACJ) as an assessment tool in educational settings

Conventional approaches to assessment involve teachers and examiners judging the quality of learners work by reference to lists of criteria or other ‘outcome’ statements. This paper explores a quite different method of assessment using ‘Adaptive Comparative Judgement’ (ACJ) that was developed within a research project at Goldsmiths University of London between 2004 and 2010. The method was developed into a tool that enabled judges to distinguish better/worse performances not by allocating numbers through mark schemes, but rather by direct, holistic, judgement. The tool was successfully deployed through a series of national and international research and development exercises. But game-changing innovations are never flaw-less first time out (Golley, Jet: Frank Whittle and the Invention of the Jet Engine, Datum Publishing, Liphook Hampshire, 2009; Dyson, Against the odds: an autobiography, Texere Publishing, Knutsford Cheshire, 2001) and a series of careful investigations resulted in a problem being identified within the workings of ACJ (Bramley, Investigating the reliability of Adaptive Comparative Judgment, Cambridge Assessment Research Report, UK, Cambridge, 2015). The issue was with the ‘adaptive’ component of the algorithm that, under certain conditions, appeared to exaggerate the reliability statistic. The problem was ‘worked’ by the software company running ACJ and a solution found. This paper reports the whole sequence of events—from the original innovation, through deployment, the emergent problem, and the resulting solution that was published at an international conference (Rangel Smith and Lynch in: PATT36 International Conference. Research & Practice in Technology Education: Perspectives on Human Capacity and Development, 2018) and subsequently deployed within a modified ACJ algorithm.