A Survey on Haptic Technologies for Mobile Augmented Reality

Augmented reality (AR) applications have gained much research and industry attention. Moreover, the mobile counterpart—mobile augmented reality (MAR) is one of the most explosive growth areas for AR applications in the mobile environment (e.g., smartphones). The technical improvements in the hardware of smartphones, tablets, and smart-glasses provide an advantage for the wide use of mobile AR in the real world and experience these AR applications anywhere. However, the mobile nature of MAR applications can limit users’ interaction capabilities, such as input and haptic feedback. In this survey, we analyze current research issues in the area of human-computer interaction for haptic technologies in MAR scenarios. The survey first presents human sensing capabilities and their applicability in AR applications. We classify haptic devices into two groups according to the triggered sense: cutaneous/tactile : touch, active surfaces, and mid-air; kinesthetic : manipulandum, grasp, and exoskeleton. Due to MAR applications’ mobile capabilities, we mainly focus our study on wearable haptic devices for each category and their AR possibilities. To conclude, we discuss the future paths that haptic feedback should follow for MAR applications and their challenges.

Experimental Evaluation on Haptic Feedback Accuracy by Using Two Self-Made Haptic Devices and One Additional Interface in Robotic Teleoperation

The goal of haptic feedback in robotic teleoperation is to enable users to accurately feel the interaction force measured at the slave side and precisely understand what is happening in the slave environment. The accuracy of the feedback force describing the error between the actual feedback force felt by a user at the master side and the measured interaction force at the slave side is the key performance indicator for haptic display in robotic teleoperation. In this paper, we evaluate the haptic feedback accuracy in robotic teleoperation via experimental method. A special interface iHandle and two haptic devices, iGrasp-T and iGrasp-R, designed for robotic teleoperation are developed for experimental evaluation. The device iHandle integrates a high-performance force sensor and a micro attitude and heading reference system which can be used to identify human upper limb motor abilities, such as posture maintenance and force application. When a user is asked to grasp the iHandle and maintain a fixed position and posture, the fluctuation value of hand posture is measured to be between 2 and 8 degrees. Based on the experimental results, human hand tremble as input noise sensed by the haptic device is found to be a major reason that results in the noise of output force from haptic device if the spring-damping model is used to render feedback force. Therefore, haptic rendering algorithms should be independent of hand motion information to avoid input noise from human hand to the haptic control loop in teleoperation. Moreover, the iHandle can be fixed at the end effector of haptic devices; iGrasp-T or iGrasp-R, to measure the output force/torque from iGrasp-T or iGrasp-Rand to the user. Experimental results show that the accuracy of the output force from haptic device iGrasp-T is approximately 0.92 N, and using the force sensor in the iHandle can compensate for the output force inaccuracy of device iGrasp-T to 0.1 N. Using a force sensor as the feedback link to form a closed-loop feedback force control system is an effective way to improve the accuracy of feedback force and guarantee high-fidelity of feedback forces at the master side in robotic teleoperation.

Design and Testing of Two Haptic Devices Based on Reconfigurable 2R Joints

This paper presents the design and testing of two haptic devices, based on reconfigurable 2R joints: an active 2R spherical mechanism-based joint and a differential gear-based joint. Based on our previous works, in which the design and kinematic analysis of both reconfigurable joints were developed, the experimental setup and the various tasks intended to test the reconfigurability, precision, force feedback system and general performance, are presented herein. Two control modes for the haptic device operation are proposed and studied. The statistical analysis tools and their selection principles are described. The mechanical design of two experimental setups and the main elements are considered in detail. The Robot Operating System nodes and the topics that are used in the software component of the experimental setup are presented and explained. The experimental testing was carried out with a number of participants and the corresponding results were analyzed with the selected statistical tools. A detailed interpretation and discussion on of the results is provided.

Emotions in Virtual Reality

One of the first senses we learn about at birth is touch, and the one sense that can deepen our experience of many situations is touch. In this paper we propose the use of emotions including touch within virtual reality (VR) to create a simulated closeness that currently can only be achieved with in-person interactions and communications. With the simulation of nonverbal cues, we can enhance a conversation or interaction in VR. Using haptic devices to deliver the simulation of touch between users via sensors and machine learning for emotion recognition based on data collected; all working towards simulated closeness in communication despite distance or being in VR. We present a direction for further research on how to simulate inperson communication within VR with the use of emotion recognition and touch to achieve a close-to-real interaction.

Blink Rate vs Inner Self-perceived Cognitive State while using Virtual Reality Applications with Haptic Devices

On average, humans blink between eight and 21 times per minute while resting. Eye actions are influenced by external and internal stimuli. This can be utilized to measure internal cognition specifically focus and attention while performing tasks. In this experiment, an individual’s self-perceived focus, attention and interaction level is collected and then correlated with eye blinks rate. The subjects are observed while performing experiments using haptic devices in a virtual environment. The experiment was modulated based on network quality and hardware quality to further quantify the effect of each scenario on subjects’ intrinsic and extrinsic focus indicators. The experiment quantified the blinks and formulated a correlation between subjects’ own perception of the event using statistical analysis. It can be found that there is an acceptable correlation between certain indicators, network quality, fatigue, stress, focus and enjoyability of the experiments. A good quality hardware and network did enhance the experience in all the subjects indicating a need for enhanced services for haptic and immersed mixed reality activities directly affecting the cognition especially in education tools. It can be inferred that eye blink rate can be used as an additional tool for measuring the cognition of individuals experience using haptic and virtual reality tools.

The influence of tactile information on the human evaluation of tactile properties

Virtual technologies such as haptic devices and virtual try-ons have been developed to bring more certainty to the non-touch shopping experience; however, they are still no substitute for the in-person experience. In order to resolve the current limitations of haptic technology, it is necessary to carry out fundamental research on the ways in which humans perceive and discern different tactile properties. This study investigated how vision and physical touch affect the evaluation of the tactile properties of knitwear and discovered factors that affect tactile evaluation in a non-touch environment. The result of this study proved that humans can perceive tactile properties similarly when they are able to physically touch the fabric, whether their vision is obstructed or not. However, participants were unable to accurately perceive the tactile properties of knitwear when they evaluated fabrics using only visual materials, especially stretchiness and flexibility. It is confirmed that a surface haptic experience could increase the accuracy of stretchiness and flexibility evaluations, but it did not help in the evaluations of fabric thickness and heaviness. Findings from interviews suggested that the shape, width and number of folds, density, and thickness of the yarn are all major factors that influence the tactile perception of knitwear when participants could only evaluate properties through on-screen visual materials. Findings from this study contributes to the improvement of the consumer experience in the contact-free shopping environment and can be used as a fundamental guide to human perception of clothing, which can support technologies such as haptic devices.

Can Haptic Stimulation Enhance Music Perception in Hearing-Impaired Listeners?

Cochlear implants (CIs) have been remarkably successful at restoring hearing in severely-to-profoundly hearing-impaired individuals. However, users often struggle to deconstruct complex auditory scenes with multiple simultaneous sounds, which can result in reduced music enjoyment and impaired speech understanding in background noise. Hearing aid users often have similar issues, though these are typically less acute. Several recent studies have shown that haptic stimulation can enhance CI listening by giving access to sound features that are poorly transmitted through the electrical CI signal. This “electro-haptic stimulation” improves melody recognition and pitch discrimination, as well as speech-in-noise performance and sound localization. The success of this approach suggests it could also enhance auditory perception in hearing-aid users and other hearing-impaired listeners. This review focuses on the use of haptic stimulation to enhance music perception in hearing-impaired listeners. Music is prevalent throughout everyday life, being critical to media such as film and video games, and often being central to events such as weddings and funerals. It represents the biggest challenge for signal processing, as it is typically an extremely complex acoustic signal, containing multiple simultaneous harmonic and inharmonic sounds. Signal-processing approaches developed for enhancing music perception could therefore have significant utility for other key issues faced by hearing-impaired listeners, such as understanding speech in noisy environments. This review first discusses the limits of music perception in hearing-impaired listeners and the limits of the tactile system. It then discusses the evidence around integration of audio and haptic stimulation in the brain. Next, the features, suitability, and success of current haptic devices for enhancing music perception are reviewed, as well as the signal-processing approaches that could be deployed in future haptic devices. Finally, the cutting-edge technologies that could be exploited for enhancing music perception with haptics are discussed. These include the latest micro motor and driver technology, low-power wireless technology, machine learning, big data, and cloud computing. New approaches for enhancing music perception in hearing-impaired listeners could substantially improve quality of life. Furthermore, effective haptic techniques for providing complex sound information could offer a non-invasive, affordable means for enhancing listening more broadly in hearing-impaired individuals.

Haptic Devices Based on Real-Time Dynamic Models of Multibody Systems

Multibody modeling of mechanical systems can be applied to various applications. Human-in-the-loop interfaces represent a growing research field, for which increasingly more devices include a dynamic multibody model to emulate the system physics in real-time. In this scope, reliable and highly dynamic sensors, to both validate those models and to measure in real-time the physical system behavior, have become crucial. In this paper, a multibody modeling approach in relative coordinates is proposed, based on symbolic equations of the physical system. The model is running in a ROS environment, which interacts with sensors and actuators. Two real-time applications with haptic feedback are presented: a piano key and a car simulator. In the present work, several sensors are used to characterize and validate the multibody model, but also to measure the system kinematics and dynamics within the human-in-the-loop process, and to ultimately validate the haptic device behavior. Experimental results for both developed devices confirm the interest of an embedded multibody model to enhance the haptic feedback performances. Besides, model parameters variations during the experiments illustrate the infinite possibilities that such model-based configurable haptic devices can offer.