Implementing high degree automation in future air traffic control (ATC) systems will be crucial for coping with increased air traffic demand and maintaining safety. However, issues associated with the passive monitoring role assumed by operators in these systems continue to be of concern. Passive monitoring can lead to a range of human operator performance problems when overseeing automation. The performance cost when human operators are placed in a passive monitoring role has been conceptualized as the out-of-the-loop (OOTL) performance problem: where adding more automation to a system makes it less likely that the operator will notice an automation failure and intervene appropriately (Endsley & Kiris, 1995). The OOTL performance problem has been attributed to numerous factors including vigilance decrements, fatigue, task disengagement, and poor situation awareness. This study tested two different approaches to addressing the OOTL performance problem associated with high degree automation in a simulation of en-route ATC (ATC-labAdvanced; Fothergill, Loft, & Neal, 2009). Following a 60-min training and practice session, 115 university student participants completed two 30-min ATC scenarios; one under manual control and one where they supervised high degree automation (counterbalanced order). The automation performed all acceptances for aircraft entering the sector of controlled airspace, handed off all departing aircraft, and resolved all conflicts between aircraft pairs that would otherwise have violated the minimum safe separation standards (except for a single automation failure event). Participants were instructed that the automation was highly reliable, but not infallible. The first aim was to confirm that while high degree automation can reduce workload, it can also lead to increased task disengagement and fatigue when compared to manual control. Furthermore, to determine how well participants supervised the automation, the conflict detection automation failed once late in the automation scenario. This failure involved two aircraft violating the minimum lateral and vertical separation standard and being missed by the automation. We expected to find that participants would fail to detect this conflict more often, or be slower to detect it, when under automation conditions, compared to a comparable conflict event presented when under manual control. Our second aim was to investigate whether these costs of automation could be ameliorated by techniques designed to improve task engagement. Participants were assigned to one of three automation conditions, including automation with (1) no acknowledgements, (2) acknowledgments, or (3) queries. In the no acknowledgements condition, automation failure monitoring was the only task performed. In the acknowledgements condition, similar to Pop et al. (2012), participants were additionally instructed to click to acknowledge each automated action, thereby potentially improving engagement by adding an active component to an otherwise passive monitoring task. In the queries condition, participants were queried regarding the past, present, and future state of aircraft on the display. The goal was to help participants maintain an accurate mental model (aka. situation awareness) when using automation. We found that automation reduced workload, increased disengagement and fatigue, and impaired detection of a single conflict detection failure event compared to manual task performance. Consistent with previous research, this shows that as a higher degree of automation is added to a system, it becomes less likely that the operator will notice automation failures and intervene appropriately (e.g. Pop et al., 2012). The first intervention tested whether adding automation acknowledgement requirements to the task made it easier for participants to detect and resolve a single automation failure event. The results showed that there was no difference between automation with and without acknowledgement requirements on workload, task disengagement, fatigue, and the detection of the automation failure event. The second intervention tested whether adding queries regarding aircraft on the display would improve failure detection performance. The queries intervention successfully reduced task disengagement and trended towards reducing fatigue, while workload was maintained at a level similar to that of manual control. These findings suggest that the manipulation successfully reduced some of the subjective deficits associated with the passive monitoring of automation. However, there was a significant cost to participants’ ability to detect and resolve the automation failure event relative to manual performance, where half the participants in the queries condition missed the automation failure entirely, compared to 25% in the no queries condition. Response times to detect the failure event were also considerably longer when queries were included compared to no queries. One explanation is that the queries condition may have been engaging to the point of distraction. This is supported by qualitative information provided by participants, where 40% mentioned that they found the queries to be distracting. Future studies may wish to examine the effectiveness of auditory queries instead of visual queries, potentially with verbal instead of typed responses. This may allow queries to reduce task disengagement and fatigue while potentially improving participants’ ability to intervene to automation failures.
