Operator Performance of the Digital Setup Fabrication for Orthodontic–Orthognathic Treatment: An Explorative Study

The purpose of this study was to explore the operator performance of the fabrication of digital orthodontic setups integrated into cone beam computed tomography (CBCT) scans. Fifteen patients who underwent a combined orthodontic–orthognathic surgical treatment were included. The pre-treatment digital dental models and CBCT scans were fused, and four operators made virtual setups twice for all patients. Differences between the virtual setups were calculated by recording tooth crown movement from the pre-treatment model to the virtual setup. To examine performance, Pearson’s correlation coefficients, duplicate measurement errors, and inter-operator differences were calculated. For intra-operator performance, correlation values varied among tooth types, with mean correlation values from 0.66 to 0.83 for the maxilla and 0.70 to 0.83 for the mandible. For inter-operator performance, mean correlation values varied from 0.40 to 0.87 for the maxilla and from 0.44 to 0.80 for the mandible. Rotational mean differences exceeded the range of clinical acceptance (>2 degrees) at 18% for the maxilla and 20.8% for the mandible, and translational mean differences exceeded the range of clinical acceptance (0.6 mm) at 9.7% and 26% for the maxilla and mandible, respectively. The intra- and inter-operator performance of digital orthodontic setup construction for virtual three-dimensional orthognathic planning shows significant errors.

Automated Alert Failures and their Impact on Operator Performance and Trust

Despite its rapid advancement, automation remains vulnerable to system failures. The reliability of automation may impact users’ trust and how they interact with it. Additionally, the type of error can uniquely redirect user behavior. This study investigated how reliability and error type impact operator trust and monitoring performance. Participants completed a monitoring task at either 50% or 90% reliability, experiencing either misses or false alarms from an automated alert system. It was hypothesized that automation reliability would impact trust, while error type would also impact reliance and compliance behaviors. Results indicated that misses had a greater impact on monitoring performance than false alarms, while reliability did not influence performance. Trust was not influenced by reliability or error type and showed no relationship with performance measures. These results can help further clarify the way automation failures shape how humans interact with automation and inform the design of future automated systems.

Human performance and system safety: Alternative perspectives on safety assessment

Discussion Panel Abstract: The recent Boeing 737MAX accidents crystalized for the public the complexity of anticipating system and operator performance and developing a system design that prevents catastrophic outcomes. The operational situations, progression of flightcrew actions, and system behaviors that led to the two accidents had not been anticipated by the manufacturer or the regulator. These accidents were only the most recent examples of our failure to anticipate and manage operational complexities and operator performance. The art and science of human factors has yet to perfect risk assessment (or safety assessment) for complex systems. In the not-so-distant past, system risk assessment made estimates of human error probabilities (HEPs) for specific operational tasks, which were combined with estimated equipment failure rates to produce an overall risk estimate. Indeed, these Human Reliability Analysis (HRA) techniques have evolved over decades and are still being developed (e.g., IDHEAS-ECA, Xing et al., 2020), partly because they satisfy the need for a simple quantitative threshold that can be used by industry and regulators: if risk probability is too high, change the design or some other aspect of operations. Through the years, there have been critiques of the HRA approach (e.g., Hollnagel, 1998) that led to revisions, such as focusing on cognitive functions instead of operator tasks, but not to the basic quantitative risk-estimation approach. Other approaches to assessing risk/safety have wandered down other paths: attempting to capture system complexity from an operator’s perspective (Roth, Mumaw, Lewis, 1994; Nuclear Regulatory Commission, 2000), or better documenting the many ways in which system operators manage complexity daily to find ways to improve their capacity (Hollnagel, Woods, & Leveson, 2006). These approaches have used different measures than HEPs; e.g., measures of operator performance, measures of interface usability/design, measures of task complexity, and the analysis of system constraints. In this panel, we offer different perspectives on risk/safety assessment as it relates to operator performance in complex systems. Foundational to assessment is deciding the nature of safety and the role of operator performance. Another important question is, as you move away from simple quantitative measures, how do you establish safety thresholds? That is, what guidance can we give to industry and regulators regarding how to measure safety and how to decide that action is required on the basis of safety.

Should We Just Let the Machines Do It? The Benefit and Cost of Action Recommendation and Action Implementation Automation

Objective To examine the effects of action recommendation and action implementation automation on performance, workload, situation awareness (SA), detection of automation failure, and return-to-manual performance in a submarine track management task. Background Theory and meta-analytic evidence suggest that with increasing degrees of automation (DOA), operator performance improves and workload decreases, but SA and return-to-manual performance declines. Method Participants monitored the location and heading of contacts in order to classify them, mark their closest point of approach (CPA), and dive when necessary. Participants were assigned either no automation, action recommendation automation, or action implementation automation. An automation failure occurred late in the task, whereby the automation provided incorrect classification advice or implemented incorrect classification actions. Results Compared to no automation, action recommendation automation benefited automated task performance and lowered workload, but cost nonautomated task performance. Action implementation automation resulted in perfect automated task performance (by default) and lowered workload, with no costs to nonautomated task performance, SA, or return-to-manual performance compared to no automation. However, participants provided action implementation automation were less likely to detect the automation failure compared to those provided action recommendations, and made less accurate classifications immediately after the automation failure, compared to those provided no automation. Conclusion Action implementation automation produced the anticipated benefits but also caused poorer automation failure detection. Application While action implementation automation may be effective for some task contexts, system designers should be aware that operators may be less likely to detect automation failures and that performance may suffer until such failures are detected.

Where Two Ends Meet: Operator and Stakeholder Perceptions of Procedures

The differences between ‘work as imagined’ (WAI) and ‘work as done’ (WAD) reflect theoretically pervasive and well-known barriers to the examination of human performance at work. Due to the dynamic and situational nature of work, the idealized performance reflected in procedures is not always done as prescribed. The identification and examination of this gap and the nature of these deviations are imperative for high-risk industries. The present study used conventional content analysis to compare stakeholders’ performance expectations to the realities of operator performance through interviews collected at a high-risk petrochemical producer. Direct comparisons of stakeholder and operator perspectives revealed divergent expectations of how procedures are used, when they’re most useful, and reasons why operators don’t utilize the procedure amendment process. These differences could be resolved through increased collaboration between stakeholders and operators. Future research should consider collaboration interventions to bridge the gap between WAI and WAD.