Moderator and observer experiences and decision-making: Remote usability testing of a web-based empathy training portal (Preprint)

BACKGROUND COVID 2019 restrictions severely curtailed empirical endeavors that involved in-person human interaction. The pandemic also stimulated our team to embrace technology in a two-fold manner. First, we created novel technology to help us to overcome pandemic restrictions in teaching empathic communication in the traditional classroom. A web-based training portal was created for users to learn empathy in an accessible, compelling, self-directed, and interactive online environment. Second, we harnessed technology to engage in remote usability testing and data collection with prospective users of our training portal. In developing our protocol, we discovered gaps in the literature on moderator and silent observer roles and experiences in conducting remote usability testing. OBJECTIVE The aim of this paper is to share our remote moderator and silent observer experiences and their use of certain tools to capture user feedback and experiences with the app. METHODS The larger-scale project employed a quantitative and think-aloud qualitative problem-discovery usability study design. Three trained research assistants collected and utilized user feedback from eight users who were asked to complete tasks in three sessions. Each research assistant had assigned roles and were asked to qualitatively describe their roles, experiences and reactions to the usability testing protocol, and suggestions for improved techniques and strategies for conducting remote usability testing. RESULTS Major results are described in relation to the research assistant experiences with the study protocol followed by recommendations for the design of future remote testing activities as well as evidence-informed training materials for usability project personnel. CONCLUSIONS Our findings highlighted that as we move towards greater remote usability testing, we also need more comprehensive understanding of human-computer interaction and its impact on usability testing outcomes. Our team realized that accessible comprehensive web-conferencing platform to conduct remote sessions is not sufficient. Lead moderator and silent observers offered their insights and recommendations for the ongoing creation and testing of training materials for their respective roles with a focus on: online interpersonal communication skills, conducting user testing protocols, troubleshooting technology and test user issues, proficiency in web-conferencing plus behavior analysis and feedback technologies, and time management. CLINICALTRIAL None.

Usability-In-Place—Remote Usability Testing Methods for Homebound Older Adults: Rapid Literature Review

Background Technology can benefit older adults in many ways, including by facilitating remote access to services, communication, and socialization for convenience or out of necessity when individuals are homebound. As people, especially older adults, self-quarantined and sheltered in place during the COVID-19 pandemic, the importance of usability-in-place became clear. To understand the remote use of technology in an ecologically valid manner, researchers and others must be able to test usability remotely. Objective Our objective was to review practical approaches for and findings about remote usability testing, particularly remote usability testing with older adults. Methods We performed a rapid review of the literature and reported on available methods, their advantages and disadvantages, and practical recommendations. This review also reported recommendations for usability testing with older adults from the literature. Results Critically, we identified a gap in the literature—a lack of remote usability testing methods, tools, and strategies for older adults, despite this population’s increased remote technology use and needs (eg, due to disability or technology experience). We summarized existing remote usability methods that were found in the literature as well as guidelines that are available for conducting in-person usability testing with older adults. Conclusions We call on the human factors research and practice community to address this gap to better support older adults and other homebound or mobility-restricted individuals.

USER EVALUATION OF INTERACTIVE THEMATIC 3D CITY MODELS – APPLICATION OF ASYNCHRONOUS REMOTE TESTING METHOD

Asynchronous remote usability testing is a method based on a software platform used to automatically record test participants' activities when they interact with a given product in their natural environment, for example, at home. This method has been frequently used in previous decades in web design and mobile application development but has rarely been utilised in geovisualization. The importance of remote usability testing has rapidly increased in 2021 due to the COVID-19 pandemic. The 3DmoveR (3D Movement and Interaction Recorder) application was used for asynchronous remote testing presented in this paper. 3DmoveR is a research tool designed for user testing of interactive 3D visualizations in web browsers using open technologies such as PHP, JavaScript, and the Three.js library. This study focuses on an evaluation of interactive 3D city models presenting thematic information expressed by colour scale. An experiment was designed as a within-subject study consisting of two simple questionnaires, a training task and six experimental trials. Finding a building of a given category (depicted as building colour) within an interactive 3D city model was used as the experimental task. Speed and accuracy of user performances were recorded, as well as user strategy, subjective evaluations, and possible intervening variables. The results were recorded from 110 participants, where 76 of them were correct and analysed further. It can be concluded that the tested colour scale (based on the Energy Performance Certificate) was not entirely appropriate. We further analysed and discussed intervening variables that may affect remote usability testing of 3D visualizations.

Usability-In-Place—Remote Usability Testing Methods for Homebound Older Adults: Rapid Literature Review (Preprint)

BACKGROUND Technology can benefit older adults in many ways, including by facilitating remote access to services, communication, and socialization for convenience or out of necessity when individuals are homebound. As people, especially older adults, self-quarantined and sheltered in place during the COVID-19 pandemic, the importance of usability-in-place became clear. To understand the remote use of technology in an ecologically valid manner, researchers and others must be able to test usability remotely. OBJECTIVE Our objective was to review practical approaches for and findings about remote usability testing, particularly remote usability testing with older adults. METHODS We performed a rapid review of the literature and reported on available methods, their advantages and disadvantages, and practical recommendations. This review also reported recommendations for usability testing with older adults from the literature. RESULTS Critically, we identified a gap in the literature—a lack of remote usability testing methods, tools, and strategies for older adults, despite this population’s increased remote technology use and needs (eg, due to disability or technology experience). We summarized existing remote usability methods that were found in the literature as well as guidelines that are available for conducting in-person usability testing with older adults. CONCLUSIONS We call on the human factors research and practice community to address this gap to better support older adults and other homebound or mobility-restricted individuals.

A Proposed Web Based Real Time Brain Computer Interface (BCI) System for Usability Testing

<p class="Abstract">Hallway testing, Remote Usability testing, Expert review, Automated expert review and A/B testing are the methods commonly used for Usability testing. However, there is no reliable system that integrates Brain Computer Interface (BCI) into the testing process with focus given towards user emotion analysis using electroencephalography (EEG) signals. This paper proposes a system that would be able to identify user emotions while they are conducting usability tests and the results would be able to increase the accuracy of the usability test. In the proposed system the results of the usability test would be displayed in real time on a dashboard and a summary report can be generated for distribution.</p>