Point Cloud Hand–Object Segmentation Using Multimodal Imaging with Thermal and Color Data for Safe Robotic Object Handover

This paper presents an application of neural networks operating on multimodal 3D data (3D point cloud, RGB, thermal) to effectively and precisely segment human hands and objects held in hand to realize a safe human–robot object handover. We discuss the problems encountered in building a multimodal sensor system, while the focus is on the calibration and alignment of a set of cameras including RGB, thermal, and NIR cameras. We propose the use of a copper–plastic chessboard calibration target with an internal active light source (near-infrared and visible light). By brief heating, the calibration target could be simultaneously and legibly captured by all cameras. Based on the multimodal dataset captured by our sensor system, PointNet, PointNet++, and RandLA-Net are utilized to verify the effectiveness of applying multimodal point cloud data for hand–object segmentation. These networks were trained on various data modes (XYZ, XYZ-T, XYZ-RGB, and XYZ-RGB-T). The experimental results show a significant improvement in the segmentation performance of XYZ-RGB-T (mean Intersection over Union: 82.8% by RandLA-Net) compared with the other three modes (77.3% by XYZ-RGB, 35.7% by XYZ-T, 35.7% by XYZ), in which it is worth mentioning that the Intersection over Union for the single class of hand achieves 92.6%.

A Human-Inspired Control Strategy for Improving Seamless Robot-To-Human Handovers

One of the challenging aspects of robotics research is to successfully establish a human-like behavioural control strategy for human–robot handover, since a robotic controller is further complicated by the dynamic nature of the human response. This paper consequently highlights the development of an appropriate set of behaviour-based control for robot-to-human object handover by first understanding an equivalent human–human handover. The optimized hybrid position and impedance control was implemented to ensure good stability, adaptability and comfort of the robot in the object handover tasks. Moreover, a questionnaire technique was employed to gather information from the participants concerning their evaluations of the developed control system. The results demonstrate that the quantitative measurement of performance of the human-inspired control strategy can be considered acceptable for seamless human–robot handovers. This also provided significant satisfaction with the overall control performance in the robotic control system, in which the robot can dexterously pass the object to the receiver in a timely and natural manner without the risk of harm or injury by the robot. Furthermore, the survey responses were in agreement with the parallel test outcomes, demonstrating significant satisfaction with the overall performance of the robot–human interaction, as measured by an average rating of 4.20 on a five-point scale.

Handover Control for Human-Robot and Robot-Robot Collaboration

Modern scenarios in robotics involve human-robot collaboration or robot-robot cooperation in unstructured environments. In human-robot collaboration, the objective is to relieve humans from repetitive and wearing tasks. This is the case of a retail store, where the robot could help a clerk to refill a shelf or an elderly customer to pick an item from an uncomfortable location. In robot-robot cooperation, automated logistics scenarios, such as warehouses, distribution centers and supermarkets, often require repetitive and sequential pick and place tasks that can be executed more efficiently by exchanging objects between robots, provided that they are endowed with object handover ability. Use of a robot for passing objects is justified only if the handover operation is sufficiently intuitive for the involved humans, fluid and natural, with a speed comparable to that typical of a human-human object exchange. The approach proposed in this paper strongly relies on visual and haptic perception combined with suitable algorithms for controlling both robot motion, to allow the robot to adapt to human behavior, and grip force, to ensure a safe handover. The control strategy combines model-based reactive control methods with an event-driven state machine encoding a human-inspired behavior during a handover task, which involves both linear and torsional loads, without requiring explicit learning from human demonstration. Experiments in a supermarket-like environment with humans and robots communicating only through haptic cues demonstrate the relevance of force/tactile feedback in accomplishing handover operations in a collaborative task.