As robotic surgery gains popularity [1, 2, 3], methods for improving situational awareness during tele- operation have become an active area of research. Literature has attempted to incorporate haptic feedback displays to enhance and improve user performance. For example, Massimino et al. [4] showed that a combi- nation of vibrotactile and auditory substitutions lead to task performance (peg-in hole task) comparable to that using a force feedback. Kitagawa et al. [5] extended this approach by using visual force displays and and auditory cues, in experiments showing comparable performance in surgical tasks (knot-tying). Reiley et al. [6] used a visual force display in a teleoperated knot-tying task to demonstrate lower forces and reduced suture breakage by trainees. The above art demonstrates the need for information overlays in telerobotic surgical tasks. However, this literature also used prototype software and tools intended only for the specic experiments. By contrast, we use the Surgical Assistant Workstation (SAW) [7, 8] in development at Johns Hopkins University to create a general information overlay, and demonstrate its utility by creating a visual warning display for telerobotic surgery that detects instruments being operated outside of the eld of view of the endoscopic camera. SAW is a modular framework for rapid prototyping of new tools and methods for robotic surgery. It includes methods for image guidance, registration with pre-operative and intra-operative images, and ability to interact with the graphical objects rendered within the display with the master or slave manipulators in a teleoperation environment. The common telesurgical system in use is the da Vinci Surgical System (Intuitive Surgical Inc.). It consists of a surgeon’s console containing the two master manipulators, a patient side cart with up to four robotic arms - three for the slave instrument manipulators which can be equipped with the removable instruments and an endoscope camera manipulator connected to a high-performance stereo vision system. The da Vinci also provides a research and development application programming interface (DiMaio, et al., [9]) that streams kinematics data and system events at congurable rates of up to 100Hz. The SAW/cisst framework also contains an interface to the da Vinci API. We present an overlay architecture (Figure 1) implemented using the cisst/SAW libraries to integrate contextual procedure and system information for improving safety, and situational awareness during these delicate and complex manipulations. While the presented methods can be modied for use with any robotic system, we used our da Vinci S Surgical System (Intuitive Surgical Inc.) for the validation experiments (Figure 2) here. Results from validation experiments with 17 users and a total of 50 training sessions totaling 214350 image frames are presented.
Visual clues act as a substitute for haptic feedback in robotic surgery
