scholarly journals Egocentric Gesture Recognition Using 3D Convolutional Neural Networks for the Spatiotemporal Adaptation of Collaborative Robots

2021 ◽  
Vol 15 ◽  
Author(s):  
Dimitris Papanagiotou ◽  
Gavriela Senteri ◽  
Sotiris Manitsaris
Keyword(s):  
Gesture Recognition ◽  
Pose Estimation ◽  
Cycle Time ◽  
Assembly Line ◽  
Vision System ◽  
Human Robot Interaction ◽  
Human Operator ◽  
Physical Interaction ◽  
Collaborative Robots ◽  
Human Operators

Collaborative robots are currently deployed in professional environments, in collaboration with professional human operators, helping to strike the right balance between mechanization and manual intervention in manufacturing processes required by Industry 4.0. In this paper, the contribution of gesture recognition and pose estimation to the smooth introduction of cobots into an industrial assembly line is described, with a view to performing actions in parallel with the human operators and enabling interaction between them. The proposed active vision system uses two RGB-D cameras that record different points of view of gestures and poses of the operator, to build an external perception layer for the robot that facilitates spatiotemporal adaptation, in accordance with the human's behavior. The use-case of this work is concerned with LCD TV assembly of an appliance manufacturer, comprising of two parts. The first part of the above-mentioned operation is assigned to a robot, strengthening the assembly line. The second part is assigned to a human operator. Gesture recognition, pose estimation, physical interaction, and sonic notification, create a multimodal human-robot interaction system. Five experiments are performed, to test if gesture recognition and pose estimation can reduce the cycle time and range of motion of the operator, respectively. Physical interaction is achieved using the force sensor of the cobot. Pose estimation through a skeleton-tracking algorithm provides the cobot with human pose information and makes it spatially adjustable. Sonic notification is added for the case of unexpected incidents. A real-time gesture recognition module is implemented through a Deep Learning architecture consisting of Convolutional layers, trained in an egocentric view and reducing the cycle time of the routine by almost 20%. This constitutes an added value in this work, as it affords the potential of recognizing gestures independently of the anthropometric characteristics and the background. Common metrics derived from the literature are used for the evaluation of the proposed system. The percentage of spatial adaptation of the cobot is proposed as a new KPI for a collaborative system and the opinion of the human operator is measured through a questionnaire that concerns the various affective states of the operator during the collaboration.

scholarly journals Human-Robot Perception in Industrial Environments: A Survey

Sensors ◽  
2021 ◽  
Vol 21 (5) ◽  
pp. 1571
Author(s):  
Andrea Bonci ◽  
Pangcheng David Cen Cheng ◽  
Marina Indri ◽  
Giacomo Nabissi ◽  
Fiorella Sibona
Keyword(s):  
Mobile Robots ◽  
Human Robot Interaction ◽  
Human Detection ◽  
Mobile Manipulators ◽  
Human Beings ◽  
Collaborative Robots ◽  
Human Safety ◽  
Fixed Base ◽  
Industrial Environments ◽  
Human Operators

Perception capability assumes significant importance for human–robot interaction. The forthcoming industrial environments will require a high level of automation to be flexible and adaptive enough to comply with the increasingly faster and low-cost market demands. Autonomous and collaborative robots able to adapt to varying and dynamic conditions of the environment, including the presence of human beings, will have an ever-greater role in this context. However, if the robot is not aware of the human position and intention, a shared workspace between robots and humans may decrease productivity and lead to human safety issues. This paper presents a survey on sensory equipment useful for human detection and action recognition in industrial environments. An overview of different sensors and perception techniques is presented. Various types of robotic systems commonly used in industry, such as fixed-base manipulators, collaborative robots, mobile robots and mobile manipulators, are considered, analyzing the most useful sensors and methods to perceive and react to the presence of human operators in industrial cooperative and collaborative applications. The paper also introduces two proofs of concept, developed by the authors for future collaborative robotic applications that benefit from enhanced capabilities of human perception and interaction. The first one concerns fixed-base collaborative robots, and proposes a solution for human safety in tasks requiring human collision avoidance or moving obstacles detection. The second one proposes a collaborative behavior implementable upon autonomous mobile robots, pursuing assigned tasks within an industrial space shared with human operators.

scholarly journals An Online Robot Teaching Method using Static Hand Gestures and Poses

2021 ◽  
Author(s):  
Digang Sun ◽  
Ping Zhang ◽  
Mingxuan Chen ◽  
Jiaxin Chen
Keyword(s):  
Gesture Recognition ◽  
Pose Estimation ◽  
Hand Gesture Recognition ◽  
Human Robot Interaction ◽  
Hand Gesture ◽  
Depth Information ◽  
Hand Gestures ◽  
Teaching Mode ◽  
Hand Pose Estimation ◽  
Hand Pose

With an increasing number of robots are employed in manufacturing, a human-robot interaction method that can teach robots in a natural, accurate, and rapid manner is needed. In this paper, we propose a novel human-robot interface based on the combination of static hand gestures and hand poses. In our proposed interface, the pointing direction of the index finger and the orientation of the whole hand are extracted to indicate the moving direction and orientation of the robot in a fast-teaching mode. A set of hand gestures are designed according to their usage in humans' daily life and recognized to control the position and orientation of the robot in a fine-teaching mode. We employ the feature extraction ability of the hand pose estimation network via transfer learning and utilize attention mechanisms to improve the performance of the hand gesture recognition network. The inputs of hand pose estimation and hand gesture recognition networks are monocular RGB images, making our method independent of depth information input and applicable to more scenarios. In the regular shape reconstruction experiments on the UR3 robot, the mean error of the reconstructed shape is less than 1 mm, which demonstrates the effectiveness and efficiency of our method.

scholarly journals An Online Robot Teaching Method using Static Hand Gestures and Poses

2021 ◽  
Author(s):  
Digang Sun ◽  
Ping Zhang ◽  
Mingxuan Chen ◽  
Jiaxin Chen
Keyword(s):  
Gesture Recognition ◽  
Pose Estimation ◽  
Hand Gesture Recognition ◽  
Human Robot Interaction ◽  
Hand Gesture ◽  
Depth Information ◽  
Hand Gestures ◽  
Teaching Mode ◽  
Hand Pose Estimation ◽  
Hand Pose

With an increasing number of robots are employed in manufacturing, a human-robot interaction method that can teach robots in a natural, accurate, and rapid manner is needed. In this paper, we propose a novel human-robot interface based on the combination of static hand gestures and hand poses. In our proposed interface, the pointing direction of the index finger and the orientation of the whole hand are extracted to indicate the moving direction and orientation of the robot in a fast-teaching mode. A set of hand gestures are designed according to their usage in humans' daily life and recognized to control the position and orientation of the robot in a fine-teaching mode. We employ the feature extraction ability of the hand pose estimation network via transfer learning and utilize attention mechanisms to improve the performance of the hand gesture recognition network. The inputs of hand pose estimation and hand gesture recognition networks are monocular RGB images, making our method independent of depth information input and applicable to more scenarios. In the regular shape reconstruction experiments on the UR3 robot, the mean error of the reconstructed shape is less than 1 mm, which demonstrates the effectiveness and efficiency of our method.

Optimizing the Cycle Time of a Multi-Spindle Pick-and-Place Machine

2007 ◽  
Author(s):  
Ashok Thiagarajan ◽  
Purushothaman Damodaran ◽  
Krishnaswami Srihari
Keyword(s):  
Cycle Time ◽  
Assembly Line ◽  
Vision System ◽  
Circuit Board ◽  
Cost Ratio ◽  
Solder Paste ◽  
Component Placement ◽  
Pick And Place ◽  
Feeder Assignment ◽  
Placement Machine

A typical Printed Circuit Board (PCB) assembly line comprises of three major process steps, namely solder paste printing, component placement, and soldering. A stencil printer is typically used to deposit adequate amount of solder paste at appropriate locations on the PCB. One or more component placement (pick-and-place type) machines are then used to populate the PCB. Finally, the entire assembly is passed through an oven for establishing the solder joint. It has been widely accepted that the component placement step is usually the bottleneck. Consequently, the cycle time of the placement machine has to be reduced in order to improve the throughput of the assembly line. Placement machines are expensive and hence their benefit-cost ratio can be improved by improving their cycle time. The objective of this research was to reduce the cycle time of a pick-and-place component placement machine with multiple spindles. The pick-and-place machine chosen for this study was the bottleneck in an assembly line at an electronics manufacturing facility. The primary functions of a placement machine are (1) to pick components from feeder slots, (2) check for any defects using the vision system, and (3) to place the components at appropriate locations. Two important decisions which affect the cycle time are (1) feeder assignment (component location along the feeder rack) and (2) the component placement sequence. A heuristic was proposed to determine the feeder assignments and the component placement sequence was determined by solving a multiple Traveling Salesman Problem (mTSP). The objective of the feeder assignment problem was to minimize the distance traveled by the head along the feeder rack (during the pick cycle). The objective of the component placement sequencing problem was to minimize the distance traveled by the head over the PCB (during the placement cycle). The individual placement tours are later sequenced such that the number of nozzle changes required is minimized. The time taken to populate a board (time to pick + place + nozzle changeover) using the proposed approach was compared to the software which was supplied by the vendor of the machine. Several ‘production boards’ were chosen for this experimental study. The proposed solution approaches outperformed the solutions suggested by the machine’s software for all the ‘production boards’ experimented with. The resulting improvements in cycle times demonstrate the effectiveness of the proposed approach.

scholarly journals A Review of Intent Detection, Arbitration, and Communication Aspects of Shared Control for Physical Human–Robot Interaction

2018 ◽  
Vol 70 (1) ◽  
Author(s):  
Dylan P. Losey ◽  
Craig G. McDonald ◽  
Edoardo Battaglia ◽  
Marcia K. O'Malley
Keyword(s):  
Upper Limb ◽  
Haptic Feedback ◽  
Survey Methods ◽  
Coupled System ◽  
Human Robot Interaction ◽  
Shared Control ◽  
Human Operator ◽  
Physical Interaction ◽  
Task Execution ◽  
Robot Interaction

As robotic devices are applied to problems beyond traditional manufacturing and industrial settings, we find that interaction between robots and humans, especially physical interaction, has become a fast developing field. Consider the application of robotics in healthcare, where we find telerobotic devices in the operating room facilitating dexterous surgical procedures, exoskeletons in the rehabilitation domain as walking aids and upper-limb movement assist devices, and even robotic limbs that are physically integrated with amputees who seek to restore their independence and mobility. In each of these scenarios, the physical coupling between human and robot, often termed physical human robot interaction (pHRI), facilitates new human performance capabilities and creates an opportunity to explore the sharing of task execution and control between humans and robots. In this review, we provide a unifying view of human and robot sharing task execution in scenarios where collaboration and cooperation between the two entities are necessary, and where the physical coupling of human and robot is a vital aspect. We define three key themes that emerge in these shared control scenarios, namely, intent detection, arbitration, and feedback. First, we explore methods for how the coupled pHRI system can detect what the human is trying to do, and how the physical coupling itself can be leveraged to detect intent. Second, once the human intent is known, we explore techniques for sharing and modulating control of the coupled system between robot and human operator. Finally, we survey methods for informing the human operator of the state of the coupled system, or the characteristics of the environment with which the pHRI system is interacting. At the conclusion of the survey, we present two case studies that exemplify shared control in pHRI systems, and specifically highlight the approaches used for the three key themes of intent detection, arbitration, and feedback for applications of upper limb robotic rehabilitation and haptic feedback from a robotic prosthesis for the upper limb.

scholarly journals Semi-Supervised Joint Learning for Hand Gesture Recognition from a Single Color Image

Sensors ◽  
2021 ◽  
Vol 21 (3) ◽  
pp. 1007
Author(s):  
Chi Xu ◽  
Yunkai Jiang ◽  
Jun Zhou ◽  
Yi Liu
Keyword(s):  
Gesture Recognition ◽  
Pose Estimation ◽  
Color Image ◽  
Recognition Performance ◽  
Recognition Task ◽  
Hand Gesture Recognition ◽  
Hand Gesture ◽  
Estimation Task ◽  
Hand Pose Estimation ◽  
Hand Pose

Hand gesture recognition and hand pose estimation are two closely correlated tasks. In this paper, we propose a deep-learning based approach which jointly learns an intermediate level shared feature for these two tasks, so that the hand gesture recognition task can be benefited from the hand pose estimation task. In the training process, a semi-supervised training scheme is designed to solve the problem of lacking proper annotation. Our approach detects the foreground hand, recognizes the hand gesture, and estimates the corresponding 3D hand pose simultaneously. To evaluate the hand gesture recognition performance of the state-of-the-arts, we propose a challenging hand gesture recognition dataset collected in unconstrained environments. Experimental results show that, the gesture recognition accuracy of ours is significantly boosted by leveraging the knowledge learned from the hand pose estimation task.

scholarly journals Ethics of Corporeal, Co-present Robots as Agents of Influence: a Review

2021 ◽  
Author(s):  
AJung Moon ◽  
Shalaleh Rismani ◽  
H. F. Machiel Van der Loos
Keyword(s):  
Artificial Intelligence ◽  
Physical Environment ◽  
Ethical Issues ◽  
Human Robot Interaction ◽  
Physical Interaction ◽  
Ethical Challenges ◽  
Natural Tendency ◽  
Ai Ethics ◽  
Recent Trends ◽  
The Impact

Abstract Purpose of Review To summarize the set of roboethics issues that uniquely arise due to the corporeality and physical interaction modalities afforded by robots, irrespective of the degree of artificial intelligence present in the system. Recent Findings One of the recent trends in the discussion of ethics of emerging technologies has been the treatment of roboethics issues as those of “embodied AI,” a subset of AI ethics. In contrast to AI, however, robots leverage human’s natural tendency to be influenced by our physical environment. Recent work in human-robot interaction highlights the impact a robot’s presence, capacity to touch, and move in our physical environment has on people, and helping to articulate the ethical issues particular to the design of interactive robotic systems. Summary The corporeality of interactive robots poses unique sets of ethical challenges. These issues should be considered in the design irrespective of and in addition to the ethics of artificial intelligence implemented in them.

scholarly journals Robust hand gesture recognition using multiple shape-oriented visual cues

2021 ◽  
Vol 2021 (1) ◽  
Author(s):  
Samy Bakheet ◽  
Ayoub Al-Hamadi
Keyword(s):  
Real Time ◽  
Gesture Recognition ◽  
Pose Estimation ◽  
Depth Map ◽  
Hand Gesture Recognition ◽  
Support Vector ◽  
Hand Gesture ◽  
Hand Pose Estimation ◽  
Time Operation ◽  
Hand Pose

AbstractRobust vision-based hand pose estimation is highly sought but still remains a challenging task, due to its inherent difficulty partially caused by self-occlusion among hand fingers. In this paper, an innovative framework for real-time static hand gesture recognition is introduced, based on an optimized shape representation build from multiple shape cues. The framework incorporates a specific module for hand pose estimation based on depth map data, where the hand silhouette is first extracted from the extremely detailed and accurate depth map captured by a time-of-flight (ToF) depth sensor. A hybrid multi-modal descriptor that integrates multiple affine-invariant boundary-based and region-based features is created from the hand silhouette to obtain a reliable and representative description of individual gestures. Finally, an ensemble of one-vs.-all support vector machines (SVMs) is independently trained on each of these learned feature representations to perform gesture classification. When evaluated on a publicly available dataset incorporating a relatively large and diverse collection of egocentric hand gestures, the approach yields encouraging results that agree very favorably with those reported in the literature, while maintaining real-time operation.

Sign in / Sign up

Export Citation Format

Share Document