Recommendations for the Empirical Assessment of Human-AI Work Systems: A Contribution to AI Measurement Science

This Report is a companion to the Report titled "Requirements for the Evaluation of Human-AI Work Systems." Whereas that Report focused on the minimum necessary empirical requirements for the assessment of AI systems, this Report provides additional recommendations and technical details to assist the developers of AI systems. Recommendations are presented covering study design, research methods, measurement, statistical analyses, and online experimentation. This guidance should be applicable to all research intended to evaluate the effectivity of AI systems.

Requirements for the Empirical Assessment of Human-AI Work Systems: A Contribution to AI Measurement Science

The development of AI systems represents a significant investment. But to realize the promise of that investment, performance assessment is necessary. Empirical evaluation of Human-AI work systems must adduce convincing empirical evidence that the work method and its AI technology are learnable, usable, and useful. The theme to this Report is the notion that AI assessment must be effective but must also be efficient. Bench testing of a prototype of an AI system cannot require extensive series of experiments with complex designs. Thus, the empirical requirements that are presented in this Report involve escaping some of the constraints that are imposed in traditional laboratory research. Also, there is a recognition of new constraints that are unique to AI evaluation contexts. Empirical requirements are presented covering study design, research methods, statistical analyses, and online experimentation. The 15 requirements presented in this Report should be applicable to all research intended to evaluate the effectivity of AI systems.

Online Psychological Testing using Executables: A Case Study on Visual Memory Resolution

In psychological research, online experimentation is a valuable alternative to lab-based experimentation. However, converting existing psychological experiments to an online format can be difficult because this usually requires a translation from Python to JavaScript code. Here, we propose and validate a novel pipeline of online experimentation using executable files, which are downloadable programs that can be run by participants on their own computers. Critically, executables facilitate the transition from a lab-based behavioral experiment to online experimentation by allowing for flexible programming using well-established libraries (i.e., PsychoPy) without the need for translation across languages (i.e., from Python to JavaScript). As a case study, we paired the executable pipeline with a shape-color “conjunction” task to assess memory resolution (Li, Fukuda, and Barense, 2021), testing participants during a synchronous virtual conference room in order to maximize data quality. In a first experiment, we tested participants one-by-one (i.e., individual testing) and found that the reliability of timing and data quality was comparable to traditional lab-based experimentation, replicating a previously reported memory effect. In a second experiment, we assessed the test-retest reliability and the scalability of executables paired with virtual conferencing of multiple participants at once. We tested up to 18 participants in the same synchronous virtual conference room (i.e., concurrent testing), replicating the first experiment within 2% of original values. These results reveal that online experimentation using executables paired with concurrent testing via virtual conferencing is an efficient, reliable, and scalable alternative to browser-based online experimentation.

Online Experimentation @FEUP: Five Years of Evaluations

The Online Experimentation @FEUP lab gathers a set of experimental resources based on Augmented Reality, Virtual Reality and Haptic Systems. The design, development and implementation of those resources are guided by the following main goals: familiarizing students with the referred technologies, complement hands-on experimentation, motivate students and promote knowledge gain. A brief presentation of the online experimental activities most utilized and evaluated in the past five years in context of different undergraduate courses and at the K12 level is presented. In all the studies conducted, the strategies adopted involve pre- and post-testing to assess knowledge gain, experimental group activities, and individual response to surveys to assess student reaction. The results published in journals, conferences proceedings and book chapters are discussed.

Online Experimentation @FEUP: five years of evaluations

The Online Experimentation @FEUP lab gathers a set of experimental resources based on Augmented Reality, Virtual Reality and Haptic Systems. The design, development and implementation of those resources are guided by the following main goals: familiarizing students with the referred technologies, complement hands-on experimentation, motivate students and promote knowledge gain. A brief presentation of the online experimental activities most utilized and evaluated in the past five years in context of different undergraduate courses and at the K12 level is presented. In all the studies conducted, the strategies adopted involve pre- and post-testing to assess knowledge gain, experimental group activities, and individual response to surveys to assess student reaction. The results published in journals, conferences proceedings and book chapters are discussed.

The Influence of Rate and Accentuation on Subjective Rhythmization

The parsing of undifferentiated tone sequences into groups of qualitatively distinct elements is one of the earliest rhythmic phenomena to have been investigated experimentally (Bolton, 1894). The present study aimed to replicate and extend these findings through online experimentation using a spontaneous grouping paradigm with forced-choice response (from 1 to 12 tones per group). Two types of isochronous sequences were used: equitone sequences, which varied only with respect to signal rate (200, 550, or 950 ms interonset intervals), and accented sequences, in which accents were added every two or three tones to test the effect of induced grouping (duple vs. triple) and accent type (intensity, duration, or pitch). In equitone sequences, participants’ grouping percepts (N = 4,194) were asymmetrical and tempo-dependent, with “no grouping” and groups of four being most frequently reported. In accented sequences, slower rate, induced triple grouping, and intensity accents correlated with increases in group length. Furthermore, the probability of observing a mixed metric type—that is, grouping percepts divisible by both two and three (6 and 12)—was found to be highest in faster sequences with induced triple grouping. These findings suggest that lower-level triple grouping gives rise to binary grouping percepts at higher metrical levels.