Enhancing Interaction with Augmented Reality through Mid-Air Haptic Feedback: Architecture Design and User Feedback

Nowadays, Augmented-Reality (AR) head-mounted displays (HMD) deliver a more immersive visualization of virtual contents, but the available means of interaction, mainly based on gesture and/or voice, are yet limited and obviously lack realism and expressivity when compared to traditional physical means. In this sense, the integration of haptics within AR may help to deliver an enriched experience, while facilitating the performance of specific actions, such as repositioning or resizing tasks, that are still dependent on the user’s skills. In this direction, this paper gathers the description of a flexible architecture designed to deploy haptically enabled AR applications both for mobile and wearable visualization devices. The haptic feedback may be generated through a variety of devices (e.g., wearable, graspable, or mid-air ones), and the architecture facilitates handling the specificity of each. For this reason, within the paper, it is discussed how to generate a haptic representation of a 3D digital object depending on the application and the target device. Additionally, the paper includes an analysis of practical, relevant issues that arise when setting up a system to work with specific devices like HMD (e.g., HoloLens) and mid-air haptic devices (e.g., Ultrahaptics), such as the alignment between the real world and the virtual one. The architecture applicability is demonstrated through the implementation of two applications: (a) Form Inspector and (b) Simon Game, built for HoloLens and iOS mobile phones for visualization and for UHK for mid-air haptics delivery. These applications have been used to explore with nine users the efficiency, meaningfulness, and usefulness of mid-air haptics for form perception, object resizing, and push interaction tasks. Results show that, although mobile interaction is preferred when this option is available, haptics turn out to be more meaningful in identifying shapes when compared to what users initially expect and in contributing to the execution of resizing tasks. Moreover, this preliminary user study reveals some design issues when working with haptic AR. For example, users may be expecting a tailored interface metaphor, not necessarily inspired in natural interaction. This has been the case of our proposal of virtual pressable buttons, built mimicking real buttons by using haptics, but differently interpreted by the study participants.

Download Full-text

Multimodal Mixed Reality Impact on a Hand Guiding Task with a Holographic Cobot

Multimodal Technologies and Interaction ◽

10.3390/mti4040078 ◽

2020 ◽

Vol 4 (4) ◽

pp. 78

Author(s):

Andoni Rivera Pinto ◽

Johan Kildal ◽

Elena Lazkano

Keyword(s):

Augmented Reality ◽

Industrial Production ◽

User Study ◽

Haptic Feedback ◽

Mixed Reality ◽

Robot Arm ◽

Pick And Place ◽

Place Task ◽

Guidance Technique ◽

The Impact

In the context of industrial production, a worker that wants to program a robot using the hand-guidance technique needs that the robot is available to be programmed and not in operation. This means that production with that robot is stopped during that time. A way around this constraint is to perform the same manual guidance steps on a holographic representation of the digital twin of the robot, using augmented reality technologies. However, this presents the limitation of a lack of tangibility of the visual holograms that the user tries to grab. We present an interface in which some of the tangibility is provided through ultrasound-based mid-air haptics actuation. We report a user study that evaluates the impact that the presence of such haptic feedback may have on a pick-and-place task of the wrist of a holographic robot arm which we found to be beneficial.

Download Full-text

Review of the Augmented Reality Systems for Shoulder Rehabilitation

Information ◽

10.3390/info10050154 ◽

2019 ◽

Vol 10 (5) ◽

pp. 154 ◽

Cited By ~ 8

Author(s):

Rosanna Maria Viglialoro ◽

Sara Condino ◽

Giuseppe Turini ◽

Marina Carbone ◽

Vincenzo Ferrari ◽

...

Keyword(s):

Clinical Trials ◽

Augmented Reality ◽

Visual Feedback ◽

Haptic Feedback ◽

Tactile Feedback ◽

Audio Feedback ◽

Wearable Technologies ◽

Body Regions ◽

Head Mounted Displays ◽

Wearable Interface

Literature shows an increasing interest for the development of augmented reality (AR) applications in several fields, including rehabilitation. Current studies show the need for new rehabilitation tools for upper extremity, since traditional interventions are less effective than in other body regions. This review aims at: Studying to what extent AR applications are used in shoulder rehabilitation, examining wearable/non-wearable technologies employed, and investigating the evidence supporting AR effectiveness. Nine AR systems were identified and analyzed in terms of: Tracking methods, visualization technologies, integrated feedback, rehabilitation setting, and clinical evaluation. Our findings show that all these systems utilize vision-based registration, mainly with wearable marker-based tracking, and spatial displays. No system uses head-mounted displays, and only one system (11%) integrates a wearable interface (for tactile feedback). Three systems (33%) provide only visual feedback; 66% present visual-audio feedback, and only 33% of these provide visual-audio feedback, 22% visual-audio with biofeedback, and 11% visual-audio with haptic feedback. Moreover, several systems (44%) are designed primarily for home settings. Three systems (33%) have been successfully evaluated in clinical trials with more than 10 patients, showing advantages over traditional rehabilitation methods. Further clinical studies are needed to generalize the obtained findings, supporting the effectiveness of the AR applications.

Download Full-text

Haplets: Finger-Worn Wireless and Low-Encumbrance Vibrotactile Haptic Feedback for Virtual and Augmented Reality

Frontiers in Virtual Reality ◽

10.3389/frvir.2021.738613 ◽

2021 ◽

Vol 2 ◽

Author(s):

Pornthep Preechayasomboon ◽

Eric Rombokas

Keyword(s):

Virtual Reality ◽

Augmented Reality ◽

User Study ◽

Haptic Feedback ◽

Hand Tracking ◽

Software Framework ◽

Vibrotactile Feedback ◽

Pressure Sensitive ◽

Drawing Task ◽

Virtual And Augmented Reality

We introduce Haplets, a wearable, low-encumbrance, finger-worn, wireless haptic device that provides vibrotactile feedback for hand tracking applications in virtual and augmented reality. Haplets are small enough to fit on the back of the fingers and fingernails while leaving the fingertips free for interacting with real-world objects. Through robust physically-simulated hands and low-latency wireless communication, Haplets can render haptic feedback in the form of impacts and textures, and supplements the experience with pseudo-haptic illusions. When used in conjunction with handheld tools, such as a pen, Haplets provide haptic feedback for otherwise passive tools in virtual reality, such as for emulating friction and pressure-sensitivity. We present the design and engineering for the hardware for Haplets, as well as the software framework for haptic rendering. As an example use case, we present a user study in which Haplets are used to improve the line width accuracy of a pressure-sensitive pen in a virtual reality drawing task. We also demonstrate Haplets used during manipulation of objects and during a painting and sculpting scenario in virtual reality. Haplets, at the very least, can be used as a prototyping platform for haptic feedback in virtual reality.

Download Full-text

Static Rest Frame to Improve Postural Stability in Virtual and Augmented Reality

Frontiers in Virtual Reality ◽

10.3389/frvir.2020.582169 ◽

2021 ◽

Vol 1 ◽

Author(s):

Sharif Mohammad Shahnewaz Ferdous ◽

Tanvir Irfan Chowdhury ◽

Imtiaz Muhammad Arafat ◽

John Quarles

Keyword(s):

Augmented Reality ◽

Postural Stability ◽

Visual Cues ◽

Rest Frame ◽

User Study ◽

Postural Instability ◽

World Vision ◽

Instability Theory ◽

Head Mounted Displays ◽

Virtual And Augmented Reality

Many users have shown increased postural instability while using Head-Mounted Displays (HMDs) as HMDs block their real-world vision. People with balance impairments are especially more affected by this as they depend more on their visual cues to maintain their balance. In addition, balance is a good indication of cybersickness according to postural instability theory. In this research, we have investigated how to use additional visual cues to improve postural stability. Through conducting one user study in Virtual Reality (VR) and Augmented Reality (AR), we have studied the effect of a Static Rest Frame (SRF) on postural stability in persons with balance impairments due to Multiple Sclerosis (MS). Results indicate that an SRF significantly improves postural stability in VR and AR for users with MS. Based on these results, we propose guidelines for designing more accessible VR and AR systems for persons with balance impairments.

Download Full-text

Comparative evaluation of WIMP and immersive natural finger interaction: a user study on CAD assembly modeling

Virtual Reality ◽

10.1007/s10055-021-00543-0 ◽

2021 ◽

Author(s):

Marius Fechter ◽

Benjamin Schleich ◽

Sandro Wartzack

Keyword(s):

User Interfaces ◽

User Study ◽

Spatial Location ◽

Ease Of Use ◽

Natural Interaction ◽

Assembly Modeling ◽

3D Space ◽

Computer Mouse ◽

Study Results ◽

Assembly Procedure

AbstractVirtual and augmented reality allows the utilization of natural user interfaces, such as realistic finger interaction, even for purposes that were previously dominated by the WIMP paradigm. This new form of interaction is particularly suitable for applications involving manipulation tasks in 3D space, such as CAD assembly modeling. The objective of this paper is to evaluate the suitability of natural interaction for CAD assembly modeling in virtual reality. An advantage of the natural interaction compared to the conventional operation by computer mouse would indicate development potential for user interfaces of current CAD applications. Our approach bases on two main elements. Firstly, a novel natural user interface for realistic finger interaction enables the user to interact with virtual objects similar to physical ones. Secondly, an algorithm automatically detects constraints between CAD components based solely on their geometry and spatial location. In order to prove the usability of the natural CAD assembly modeling approach in comparison with the assembly procedure in current WIMP operated CAD software, we present a comparative user study. Results show that the VR method including natural finger interaction significantly outperforms the desktop-based CAD application in terms of efficiency and ease of use.

Download Full-text

Personalized Augmented Reality Based Tourism System: Big Data and User Demographic Contexts

Applied Sciences ◽

10.3390/app11136047 ◽

2021 ◽

Vol 11 (13) ◽

pp. 6047

Author(s):

Soheil Rezaee ◽

Abolghasem Sadeghi-Niaraki ◽

Maryam Shakeri ◽

Soo-Mi Choi

Keyword(s):

Big Data ◽

Augmented Reality ◽

User Study ◽

Spatial Information ◽

Ease Of Use ◽

Tourist Attraction ◽

Gender And Education ◽

Time Distance ◽

Data Source ◽

The Right

A lack of required data resources is one of the challenges of accepting the Augmented Reality (AR) to provide the right services to the users, whereas the amount of spatial information produced by people is increasing daily. This research aims to design a personalized AR that is based on a tourist system that retrieves the big data according to the users’ demographic contexts in order to enrich the AR data source in tourism. This research is conducted in two main steps. First, the type of the tourist attraction where the users interest is predicted according to the user demographic contexts, which include age, gender, and education level, by using a machine learning method. Second, the correct data for the user are extracted from the big data by considering time, distance, popularity, and the neighborhood of the tourist places, by using the VIKOR and SWAR decision making methods. By about 6%, the results show better performance of the decision tree by predicting the type of tourist attraction, when compared to the SVM method. In addition, the results of the user study of the system show the overall satisfaction of the participants in terms of the ease-of-use, which is about 55%, and in terms of the systems usefulness, about 56%.

Download Full-text

Improved Mutual Understanding for Human-Robot Collaboration: Combining Human-Aware Motion Planning with Haptic Feedback Devices for Communicating Planned Trajectory

Sensors ◽

10.3390/s21113673 ◽

2021 ◽

Vol 21 (11) ◽

pp. 3673

Author(s):

Stefan Grushko ◽

Aleš Vysocký ◽

Petr Oščádal ◽

Michal Vocetka ◽

Petr Novák ◽

...

Keyword(s):

User Study ◽

Haptic Feedback ◽

Fundamental Aspect ◽

Task Completion ◽

Good Efficiency ◽

Notification System ◽

Movement Data ◽

Novel Approach ◽

Machine Interface ◽

High Degree

In a collaborative scenario, the communication between humans and robots is a fundamental aspect to achieve good efficiency and ergonomics in the task execution. A lot of research has been made related to enabling a robot system to understand and predict human behaviour, allowing the robot to adapt its motion to avoid collisions with human workers. Assuming the production task has a high degree of variability, the robot’s movements can be difficult to predict, leading to a feeling of anxiety in the worker when the robot changes its trajectory and approaches since the worker has no information about the planned movement of the robot. Additionally, without information about the robot’s movement, the human worker cannot effectively plan own activity without forcing the robot to constantly replan its movement. We propose a novel approach to communicating the robot’s intentions to a human worker. The improvement to the collaboration is presented by introducing haptic feedback devices, whose task is to notify the human worker about the currently planned robot’s trajectory and changes in its status. In order to verify the effectiveness of the developed human-machine interface in the conditions of a shared collaborative workspace, a user study was designed and conducted among 16 participants, whose objective was to accurately recognise the goal position of the robot during its movement. Data collected during the experiment included both objective and subjective parameters. Statistically significant results of the experiment indicated that all the participants could improve their task completion time by over 45% and generally were more subjectively satisfied when completing the task with equipped haptic feedback devices. The results also suggest the usefulness of the developed notification system since it improved users’ awareness about the motion plan of the robot.

Download Full-text

ARETT: Augmented Reality Eye Tracking Toolkit for Head Mounted Displays

Sensors ◽

10.3390/s21062234 ◽

2021 ◽

Vol 21 (6) ◽

pp. 2234

Author(s):

Sebastian Kapp ◽

Michael Barz ◽

Sergey Mukhametov ◽

Daniel Sonntag ◽

Jochen Kuhn

Keyword(s):

Augmented Reality ◽

Eye Tracking ◽

User Interaction ◽

R Package ◽

Eye Tracker ◽

Unity 3D ◽

Accuracy And Precision ◽

Head Mounted Displays ◽

Virtual And Augmented Reality ◽

Microsoft Hololens

Currently an increasing number of head mounted displays (HMD) for virtual and augmented reality (VR/AR) are equipped with integrated eye trackers. Use cases of these integrated eye trackers include rendering optimization and gaze-based user interaction. In addition, visual attention in VR and AR is interesting for applied research based on eye tracking in cognitive or educational sciences for example. While some research toolkits for VR already exist, only a few target AR scenarios. In this work, we present an open-source eye tracking toolkit for reliable gaze data acquisition in AR based on Unity 3D and the Microsoft HoloLens 2, as well as an R package for seamless data analysis. Furthermore, we evaluate the spatial accuracy and precision of the integrated eye tracker for fixation targets with different distances and angles to the user (n=21). On average, we found that gaze estimates are reported with an angular accuracy of 0.83 degrees and a precision of 0.27 degrees while the user is resting, which is on par with state-of-the-art mobile eye trackers.

Download Full-text

Use cases and usability challenges for head-mounted displays in healthcare

Current Directions in Biomedical Engineering ◽

10.1515/cdbme-2015-0127 ◽

2015 ◽

Vol 1 (1) ◽

pp. 534-537 ◽

Cited By ~ 5

Author(s):

T. Mentler ◽

C. Wolters ◽

M. Herczeg

Keyword(s):

Augmented Reality ◽

Emergency Medical Services ◽

Information Visualization ◽

Nursing Staff ◽

Interaction Design ◽

Use Cases ◽

Design Approach ◽

User Centered Design ◽

Head Mounted Displays ◽

Mobile Context

AbstractIn the healthcare domain, head-mounted displays (HMDs) with augmented reality (AR) modalities have been reconsidered for application as a result of commercially available products and the needs for using computers in mobile context. Within a user-centered design approach, interviews were conducted with physicians, nursing staff and members of emergency medical services. Additionally practitioners were involved in evaluating two different head-mounted displays. Based on these measures, use cases and usability considerations according to interaction design and information visualization were derived and are described in this contribution.

Download Full-text

Learning Retention of Thoracic Pedicle Screw Placement Using a High-Resolution Augmented Reality Simulator With Haptic Feedback

Operative Neurosurgery ◽

10.1227/neu.0b013e31821954ed ◽

2011 ◽

Vol 69 (suppl_1) ◽

pp. ons14-ons19 ◽

Cited By ~ 24

Author(s):

Cristian J Luciano ◽

P Pat Banerjee ◽

Brad Bellotte ◽

G Michael Oh ◽

Michael Lemole ◽

...

Keyword(s):

Augmented Reality ◽

Pedicle Screw ◽

Haptic Feedback ◽

Screw Placement ◽

Pedicle Screw Placement ◽

Data Set ◽

Learning Retention ◽

Performance Accuracy ◽

Thoracic Pedicle Screw ◽

Thoracic Pedicle Screws

Abstract BACKGROUND: We evaluated the use of a part-task simulator with 3D and haptic feedback as a training tool for a common neurosurgical procedure - placement of thoracic pedicle screws. OBJECTIVE: To evaluate the learning retention of thoracic pedicle screw placement on a high-performance augmented reality and haptic technology workstation. METHODS: Fifty-one fellows and residents performed thoracic pedicle screw placement on the simulator. The virtual screws were drilled into a virtual patient's thoracic spine derived from a computed tomography data set of a real patient. RESULTS: With a 12.5% failure rate, a 2-proportion z test yielded P = .08. For performance accuracy, an aggregate Euclidean distance deviation from entry landmark on the pedicle and a similar deviation from the target landmark in the vertebral body yielded P = .04 from a 2-sample t test in which the rejected null hypothesis assumes no improvement in performance accuracy from the practice to the test sessions, and the alternative hypothesis assumes an improvement. CONCLUSION: The performance accuracy on the simulator was comparable to the accuracy reported in literature on recent retrospective evaluation of such placements. The failure rates indicated a minor drop from practice to test sessions, and also indicated a trend (P = .08) toward learning retention resulting in improvement from practice to test sessions. The performance accuracy showed a 15% mean score improvement and more than a 50% reduction in standard deviation from practice to test. It showed evidence (P = .04) of performance accuracy improvement from practice to test session.

Download Full-text