Egocentric-View Fingertip Detection for Air Writing Based on Convolutional Neural Networks

This research investigated real-time fingertip detection in frames captured from the increasingly popular wearable device, smart glasses. The egocentric-view fingertip detection and character recognition can be used to create a novel way of inputting texts. We first employed Unity3D to build a synthetic dataset with pointing gestures from the first-person perspective. The obvious benefits of using synthetic data are that they eliminate the need for time-consuming and error-prone manual labeling and they provide a large and high-quality dataset for a wide range of purposes. Following that, a modified Mask Regional Convolutional Neural Network (Mask R-CNN) is proposed, consisting of a region-based CNN for finger detection and a three-layer CNN for fingertip location. The process can be completed in 25 ms per frame for 640×480 RGB images, with an average error of 8.3 pixels. The speed is high enough to enable real-time “air-writing”, where users are able to write characters in the air to input texts or commands while wearing smart glasses. The characters can be recognized by a ResNet-based CNN from the fingertip trajectories. Experimental results demonstrate the feasibility of this novel methodology.

Real-time virtual mouse system using RGB-D images and fingertip detection

A real-time fingertip-gesture-based interface is still challenging for human–computer interactions, due to sensor noise, changing light levels, and the complexity of tracking a fingertip across a variety of subjects. Using fingertip tracking as a virtual mouse is a popular method of interacting with computers without a mouse device. In this work, we propose a novel virtual-mouse method using RGB-D images and fingertip detection. The hand region of interest and the center of the palm are first extracted using in-depth skeleton-joint information images from a Microsoft Kinect Sensor version 2, and then converted into a binary image. Then, the contours of the hands are extracted and described by a border-tracing algorithm. The K-cosine algorithm is used to detect the fingertip location, based on the hand-contour coordinates. Finally, the fingertip location is mapped to RGB images to control the mouse cursor based on a virtual screen. The system tracks fingertips in real-time at 30 FPS on a desktop computer using a single CPU and Kinect V2. The experimental results showed a high accuracy level; the system can work well in real-world environments with a single CPU. This fingertip-gesture-based interface allows humans to easily interact with computers by hand.

Real Time Vision Based Fingertip Detection Strategies

The Fingertip Detection acts a specific role in most of the vision based applications. The latest technologies like virtual reality and augmented reality actually follows this fingertip detection concept as its foundation. It is also helpful for Human Computer Interaction (HCI). So fingertip detection and tracking can be applied from games to robot control, from augmented reality to smart homes. The most important interesting field of fingertip detection is the gesture recognition related applications. In the context of interaction with the machines, gestures are the most simplest and efficient means of communication. This paper analyses the various works done in the areas of fingertip detection. A review on various real time fingertip methods is explained with different techniques and tools. Some challenges and research directions are also highlighted. Many researchers uses fingertip detection in HCI systems those have many applications in user identification, smart home etc. A comparison of results by different researchers is also included.