An adaptive human sensor framework for human–robot collaboration

Manufacturing challenges are increasing the demands for more agile and dexterous means of production. At the same time, these systems aim to maintain or even increase productivity. The challenges risen from these developments can be tackled through human–robot collaboration (HRC). HRC requires effective task distribution according to each party’s distinctive strengths, which is envisioned to generate synergetic effects. To enable a seamless collaboration, the human and robot require a mutual awareness, which is challenging, due to the human and robot “speaking” different languages as in analogue and digital. This challenge can be addressed by equipping the robot with a model of the human. Despite a range of models being available, data-driven models of the human are still at an early stage. For this purpose, this paper proposes an adaptive human sensor framework, which incorporates objective, subjective, and physiological metrics, as well as associated machine learning. Thus, it is envisioned to adapt to the uniqueness and dynamic nature of human behavior. To test the framework, a validation experiment was performed, including 18 participants, which aims to predict perceived workload during two scenarios, namely a manual and an HRC assembly task. Perceived workloads are described to have a substantial impact on a human operator’s task performance. Throughout the experiment, physiological data from an electroencephalogram (EEG), an electrocardiogram (ECG), and respiration sensor was collected and interpreted. For subjective metrics, the standardized NASA Task Load Index was used. Objective metrics included task completion time and number of errors/assistance requests. Overall, the framework revealed a promising potential towards an adaptive behavior, which is ultimately envisioned to enable a more effective HRC.

Augmented Reality and Intraoperative Navigation in Sinonasal Malignancies: A Preclinical Study

ObjectiveTo report the first use of a novel projected augmented reality (AR) system in open sinonasal tumor resections in preclinical models and to compare the AR approach with an advanced intraoperative navigation (IN) system.MethodsFour tumor models were created. Five head and neck surgeons participated in the study performing virtual osteotomies. Unguided, AR, IN, and AR + IN simulations were performed. Statistical comparisons between approaches were obtained. Intratumoral cut rate was the main outcome. The groups were also compared in terms of percentage of intratumoral, close, adequate, and excessive distances from the tumor. Information on a wearable gaze tracker headset and NASA Task Load Index questionnaire results were analyzed as well.ResultsA total of 335 cuts were simulated. Intratumoral cuts were observed in 20.7%, 9.4%, 1.2,% and 0% of the unguided, AR, IN, and AR + IN simulations, respectively (p < 0.0001). The AR was superior than the unguided approach in univariate and multivariate models. The percentage of time looking at the screen during the procedures was 55.5% for the unguided approaches and 0%, 78.5%, and 61.8% in AR, IN, and AR + IN, respectively (p < 0.001). The combined approach significantly reduced the screen time compared with the IN procedure alone.ConclusionWe reported the use of a novel AR system for oncological resections in open sinonasal approaches, with improved margin delineation compared with unguided techniques. AR improved the gaze-toggling drawback of IN. Further refinements of the AR system are needed before translating our experience to clinical practice.

Biosignal-Based Driving Skill Classification Using Machine Learning: A Case Study of Maritime Navigation

This work presents a novel approach to detecting stress differences between experts and novices in Situation Awareness (SA) tasks during maritime navigation using one type of wearable sensor, Empatica E4 Wristband. We propose that for a given workload state, the values of biosignal data collected from wearable sensor vary in experts and novices. We describe methods to conduct a designed SA task experiment, and collected the biosignal data on subjects sailing on a 240° view simulator. The biosignal data were analysed by using a machine learning algorithm, a Convolutional Neural Network. The proposed algorithm showed that the biosingal data associated with the experts can be categorized as different from that of the novices, which is in line with the results of NASA Task Load Index (NASA-TLX) rating scores. This study can contribute to the development of a self-training system in maritime navigation in further studies.

The Effect of Task Complexity on Time Estimation in the Virtual Reality Environment: An EEG Study

This paper investigated the effect of task complexity on time estimation in the virtual reality environment (VRE) using behavioral, subjective, and physiological measurements. Virtual reality (VR) is not a perfect copy of the real world, and individuals perceive time duration differently in the VRE than they do in reality. Though many researchers have found a connection between task complexity and time estimation under non-VR conditions, the influence of task complexity on time estimation in the VRE is yet unknown. In this study, twenty-nine participants performed a VR jigsaw puzzle task at two levels of task complexity. We observed that as task complexity increased, participants showed larger time estimation errors, reduced relative beta-band power at Fz and Pz, and higher NASA-Task Load Index scores. Our findings indicate the importance of controlling task complexity in the VRE and demonstrate the potential of using electroencephalography (EEG) as real-time indicators of complexity level.

P090 The Effect of Workload, Sleep Deprivation and Time of Day on Simulated Driving Performance

Sustained operations expose individuals to long work periods, which deteriorates their ability to sustain attention. Biological factors, including sleep deprivation and time of day, have been shown to play a critical role in the ability to sustain attention. However, a gap in the literature exists regarding external factors, such as workload. Therefore, the aim of this study was to investigate the combined effect of sleep deprivation, time of day, and workload on sustained attention. Twenty-one participants (18–34y, 10 F) were exposed to 62 hours of sleep deprivation within a controlled laboratory environment. Every 8 hours, sustained attention was measured using a 30-minute monotonous driving task, and subjective workload was measured using the NASA-Task Load Index (TLX). Workload, defined as time on task was assessed by splitting the drive into two 15-minute loops. A mixed model ANOVA revealed significant main effects of day (sleep deprivation) and time of day on lane deviation, number of crashes, speed deviation and time outside the safe zone (all p<.001). There was a significant main effect of workload (time on task) on lane deviation (p=.042), indicating that a longer time on task resulted in greater lane deviation. NASA-TLX scores significantly increased with sleep deprivation (p<.001), indicating that subjective workload increased with sleep loss even though the task remained constant. Workload, sleep deprivation and time of day produced a deterioration in sustained attention. With this, countermeasures that not only consider sleep deprivation and time of day, but also workload (time on task) can be considered.

Ageing drivers’ mental workload in real-time driving tasks based on subjective and objective measures

The ageing drivers’ population is increasing rapidly, and they are exposed to disabilities due to degenerative processes, thus affecting their driving performance. The main objective of this study is to determine the mental workload of ageing drivers, while the second objective is to compare the mental workload between ageing drivers and control group. The methodology consisted of on-the-road experimental driving tasks that comprised three levels of situation complexity. The NASA-Task Load Index (NASA-TLX) and electroencephalogram (EEG) were measured on 30 drivers. The NASA-TLX scores revealed that the ageing drivers’ mean physical demand score was the highest compared to others in moderately complex situation and very complex situation, scoring 37.25 and 43.50, respectively. Meanwhile, for electroencephalogram signals’ fluctuation, results showed that situation complexity had significant effects on RPθ and RPα of channel locations FZPZ and O1O2. There was a significant difference in the weighted workload scores for the ageing drivers and control group in simple situation, while there was no significant difference found in RPθ and RPα bands at all channel locations. The findings would be beneficial as a guideline for designers, manufacturers, developers, and policy makers in designing better driving environment for ageing drivers.

Testing silicone digit extensions as a way to suppress natural sensation to evaluate supplementary tactile feedback

Dexterous use of the hands depends critically on sensory feedback, so it is generally agreed that functional supplementary feedback would greatly improve the use of hand prostheses. Much research still focuses on improving non-invasive feedback that could potentially become available to all prosthesis users. However, few studies on supplementary tactile feedback for hand prostheses demonstrated a functional benefit. We suggest that confounding factors impede accurate assessment of feedback, e.g., testing non-amputee participants that inevitably focus intently on learning EMG control, the EMG’s susceptibility to noise and delays, and the limited dexterity of hand prostheses. In an attempt to assess the effect of feedback free from these constraints, we used silicone digit extensions to suppress natural tactile feedback from the fingertips and thus used the tactile feedback-deprived human hand as an approximation of an ideal feed-forward tool. Our non-amputee participants wore the extensions and performed a simple pick-and-lift task with known weight, followed by a more difficult pick-and-lift task with changing weight. They then repeated these tasks with one of three kinds of audio feedback. The tests were repeated over three days. We also conducted a similar experiment on a person with severe sensory neuropathy to test the feedback without the extensions. Furthermore, we used a questionnaire based on the NASA Task Load Index to gauge the subjective experience. Unexpectedly, we did not find any meaningful differences between the feedback groups, neither in the objective nor the subjective measurements. It is possible that the digit extensions did not fully suppress sensation, but since the participant with impaired sensation also did not improve with the supplementary feedback, we conclude that the feedback failed to provide relevant grasping information in our experiments. The study highlights the complex interaction between task, feedback variable, feedback delivery, and control, which seemingly rendered even rich, high-bandwidth acoustic feedback redundant, despite substantial sensory impairment.

Mental Workload of Local vs Remote Operator in Human-Machine Interaction Case Study

Since late 2019, a novel Coronavirus disease 2019 (COVID-19) has spread globally. As a result, businesses were forced to send their workforce into remote working, wherever possible. While research in this area has seen an increase in studying and developing technologies that allow and support such remote working style, not every sector is currently prepared for such a transition. Especially the manufacturing sector has faced challenges in this regard. In this paper, the mental workload of two groups of participants is studied during a human-robot interaction task. Participants were asked to bring a robotised cell used in a dispensing task to full production by tuning system parameters. After the experiment, a self-assessment of the participants’ perceived mental workload using the NASA Task Load Index (NASA-TLX) was used. The results show that remote participants tend to have lower perceived workload compared to the local participants.

An Adaptive Human Sensor Framework for Human-Robot Collaboration

Manufacturing challenges are increasing the demands for more agile and dexterous means of production. At the same time, these systems aim to maintain or even increase productivity. The challenges risen from these developments can be tackled through Human-Robot Collaboration (HRC). HRC requires effective task distribution according to each parties’ distinctive strengths, which is envisioned to generate synergetic effects. To enable a seamless collaboration, the human and robot require a mutual awareness, which is challenging, due to the human and robot “speaking” different languages as in analogue and digital. Thus, this challenge can be addressed by equipping the robot with a model of the human. Despite a range of models being available, data-driven models of the human are still at an early stage. This paper proposes an adaptive human sensor framework, which incorporates objective, subjective, and physiological metrics, as well as associated Machine Learning. Thus, it is envisioned to adapt to the uniqueness and dynamic nature of human behavior. To test the framework, a validation experiment was performed, including 18 participants, which aims to predict Perceived Workload during two scenarios, namely a manual and an HRC assembly task. Perceived Workloads are described to have a substantial impact on a human operator’s task performance. Throughout the experiment physiological data from an electroencephalogram (EEG), an electrocardiogram (ECG), and respiration sensor was collected and interpreted. For subjective metrics, the standardized NASA Task Load Index was used. Objective metrics included task completion time and number of errors/assistance requests. Overall, the framework revealed a promising potential towards an adaptive behavior, which is ultimately envisioned to enable a more effective HRC.