Mixed Reality-Enhanced Intuitive Teleoperation with Hybrid Virtual Fixtures for Intelligent Robotic Welding

This paper presents an integrated scheme based on a mixed reality (MR) and haptic feedback approach for intuitive and immersive teleoperation of robotic welding systems. By incorporating MR technology, the user is fully immersed in a virtual operating space augmented by real-time visual feedback from the robot working space. The proposed robotic tele-welding system features imitative motion mapping from the user’s hand movements to the welding robot motions, and it enables the spatial velocity-based control of the robot tool center point (TCP). The proposed mixed reality virtual fixture (MRVF) integration approach implements hybrid haptic constraints to guide the operator’s hand movements following the conical guidance to effectively align the welding torch for welding and constrain the welding operation within a collision-free area. Onsite welding and tele-welding experiments identify the operational differences between professional and unskilled welders and demonstrate the effectiveness of the proposed MRVF tele-welding framework for novice welders. The MRVF-integrated visual/haptic tele-welding scheme reduced the torch alignment times by 56% and 60% compared to the MRnoVF and baseline cases, with minimized cognitive workload and optimal usability. The MRVF scheme effectively stabilized welders’ hand movements and eliminated undesirable collisions while generating smooth welds.

A Digital Twin Approach for Contextual Assistance for Surgeons During Surgical Robotics Training

Minimally invasive robotic surgery copes with some disadvantages for the surgeon of minimally invasive surgery while preserving the advantages for the patient. Most commercially available robotic systems are telemanipulated with haptic input devices. The exploitation of the haptics channel, e.g., by means of Virtual Fixtures, would allow for an individualized enhancement of surgical performance with contextual assistance. However, it remains an open field of research as it is non-trivial to estimate the task context itself during a surgery. In contrast, surgical training allows to abstract away from a real operation and thus makes it possible to model the task accurately. The presented approach exploits this fact to parameterize Virtual Fixtures during surgical training, proposing a Shared Control Parametrization Engine that retrieves procedural context information from a Digital Twin. This approach accelerates a proficient use of the robotic system for novice surgeons by augmenting the surgeon’s performance through haptic assistance. With this our aim is to reduce the required skill level and cognitive load of a surgeon performing minimally invasive robotic surgery. A pilot study is performed on the DLR MiroSurge system to evaluate the presented approach. The participants are tasked with two benchmark scenarios of surgical training. The execution of the benchmark scenarios requires basic skills as pick, place and path following. The evaluation of the pilot study shows the promising trend that novel users profit from the haptic augmentation during training of certain tasks.

Collaborative Robotic Assistant Platform for Endonasal Surgery: Preliminary In-Vitro Trials

Endonasal surgery is a minimally invasive approach for the removal of pituitary tumors (sarcomas). In this type of procedure, the surgeon has to complete the surgical maneuvers for sarcoma resection with extreme precision, as there are many vital structures in this area. Therefore, the use of robots for this type of intervention could increase the success of the intervention by providing accurate movements. Research has focused on the development of teleoperated robots to handle a surgical instrument, including the use of virtual fixtures to delimit the working area. This paper aims to go a step further with a platform that includes a teleoperated robot and an autonomous robot dedicated to secondary tasks. In this way, the aim is to reduce the surgeon’s workload so that he can concentrate on his main task. Thus, the article focuses on the description and implementation of a navigator that coordinates both robots via a force/position control. Finally, both the navigation and control scheme were validated by in-vitro tests.

Virtual Fixture Generation for Task Planning with Complex Geometries

Many robotic processes require the system to maintain a tool's orientation and distance from a surface. To do so, researchers often use Virtual Fixtures (VFs) to either guide the robot along a path or forbid it from leaving the workspace. Previous efforts relied on volumetric primitives (planes, cylinders, etc.) or raw sensor data to define VFs. However, those approaches only work for a small subset of real-world objects. Extending this approach is complicated not only by VF generation but also generalizing user traversal of the VF to command a robot trajectory remotely. In this work, we present the concept of Task VFs, which convert layers of point cloud based Guidance VF into a bidirectional graph structure and pair it with a Forbidden Region VF. These VFs are hardware-agnostic and can be generated from virtually any source data, including from parametric objects (superellipsoids, supertoroids, etc.), meshes (including from CAD), and real-time sensor data for open-world scenarios. We address surface convexity and concavity since these and distance to the task surface determine the size and resolution of VF layers. This paper then presents the Manipulator-to-Task Transform Tool for Task VF visualization and to limit human-robot interaction ambiguities. Testing confirmed generation success, and users performed spatially discrete experiments to evaluate Task VF usability complex geometries, which showed their interpretability. The Manipulator-to-Task Transform Tool applies many robotic applications, including collision avoidance, process design, training, task definition, etc. for virtually any geometry.

Promoting mutual adaptation in haptic negotiation using adaptive virtual fixture

Purpose To promote the intuitiveness of collaborative tasks, the negotiation ability of humans with each other has inspired a large amount of studies aimed at reproducing the capacity in physical human-robot interaction (pHRI). This paper aims to promote mutual adaptation in negotiation when both parties possess incomplete information. Design/methodology/approach This paper introduces virtual fixtures into the traditional negotiation mechanism, locally regulating tracking trajectory and impedance parameters in the negotiating phase until the final plan integrates bilateral intentions well. In the strategy, robots convey its task information to humans and offer groups of guide plans for them to choose, on the premise of maximizing the robot’s own profits. Findings Compared with traditional negotiation strategies, humans adapt to robots easily and show lower cognitive load in the method, while the satisfied plan shows better performance for the whole human-robot system. Originality/value In this study, this paper proposes a novel negotiation strategy to facilitate the mutual adaptation of humans and robots in complicated shared tasks, especially when both parties possess incomplete information of tasks.

Telelocomotion—Remotely Operated Legged Robots

Teleoperated systems enable human control of robotic proxies and are particularly amenable to inaccessible environments unsuitable for autonomy. Examples include emergency response, underwater manipulation, and robot assisted minimally invasive surgery. However, teleoperation architectures have been predominantly employed in manipulation tasks, and are thus only useful when the robot is within reach of the task. This work introduces the idea of extending teleoperation to enable online human remote control of legged robots, or telelocomotion, to traverse challenging terrain. Traversing unpredictable terrain remains a challenge for autonomous legged locomotion, as demonstrated by robots commonly falling in high-profile robotics contests. Telelocomotion can reduce the risk of mission failure by leveraging the high-level understanding of human operators to command in real-time the gaits of legged robots. In this work, a haptic telelocomotion interface was developed. Two within-user studies validate the proof-of-concept interface: (i) The first compared basic interfaces with the haptic interface for control of a simulated hexapedal robot in various levels of traversal complexity; (ii) the second presents a physical implementation and investigated the efficacy of the proposed haptic virtual fixtures. Results are promising to the use of haptic feedback for telelocomotion for complex traversal tasks.