A Survey on Haptic Technologies for Mobile Augmented Reality

Augmented reality (AR) applications have gained much research and industry attention. Moreover, the mobile counterpart—mobile augmented reality (MAR) is one of the most explosive growth areas for AR applications in the mobile environment (e.g., smartphones). The technical improvements in the hardware of smartphones, tablets, and smart-glasses provide an advantage for the wide use of mobile AR in the real world and experience these AR applications anywhere. However, the mobile nature of MAR applications can limit users’ interaction capabilities, such as input and haptic feedback. In this survey, we analyze current research issues in the area of human-computer interaction for haptic technologies in MAR scenarios. The survey first presents human sensing capabilities and their applicability in AR applications. We classify haptic devices into two groups according to the triggered sense: cutaneous/tactile : touch, active surfaces, and mid-air; kinesthetic : manipulandum, grasp, and exoskeleton. Due to MAR applications’ mobile capabilities, we mainly focus our study on wearable haptic devices for each category and their AR possibilities. To conclude, we discuss the future paths that haptic feedback should follow for MAR applications and their challenges.

Virtual Bronchoscopy-Guided Transbronchial Biopsy Simulation Using a Head-Mounted Display: A New Style of Flexible Bronchoscopy

Background/need. The increases in reference images and information during bronchoscopy using virtual bronchoscopic navigation (VBN) and fluoroscopy has potentially created the need for support using a head-mounted display (HMD) because bronchoscopists feel difficulty to see displays that are at a distance from them and turn their head and body in various directions. Methodology and device description. The binocular see-through Moverio BT-35E Smart Glasses can be connected via a high-definition multimedia interface and have a 720p high-definition display. We developed a system that converts fluoroscopic (live and reference), VBN, and bronchoscopic image signals through a converter and references them using the Moverio BT-35E. Preliminary results. We performed a virtual bronchoscopy-guided transbronchial biopsy simulation using the system. Four experienced pulmonologists performed a simulated bronchoscopy of 5 cases each with the Moverio BT-35E glasses, using bronchoscopy training model. For all procedures, the bronchoscope was advanced successfully into the target bronchus according to the VBN image. None of the operators reported eye or body fatigue during or after the procedure. Current status. This small-scale simulation study suggests the feasibility of using a HMD during bronchoscopy. For clinical use, it is necessary to evaluate the safety and usefulness of the system in larger clinical trials in the future.

Technology acceptance drivers for AR smart glasses in the middle east: A quantitative study

This study aims to establish Middle East users' perspectives on the major factors that impact their decision to adopt Augmented Reality AR smart glasses (ARSG). Thus, an online questionnaire was designed and sent directly to the respondents, and 584 valid data points were collected from individuals living in the Middle East. The data were analyzed using Pearson correlations and Exploratory Factor Analysis (EFA) techniques using SPSS. Eleven hypotheses were tested using Multiple Regression analysis, where seven independent variables out of eleven were confirmed to have a significant impact on the perceived adoption of ARSG. The results indicate that four of the independent variables including Pre-Market Knowledge, Image, Own privacy and Technology innovativeness show the significant impact on ARSG adoption at the 1% significant level. In addition, the results indicate that three of the social and technological factors include Perceived Ease of use, Perceived usefulness and Other's privacy show the significant effect on ARSG adoption at the 5% significant level. Among the 7 social and technological factors, the results suggest that technology innovation expresses the strongest effect on ARSG adoption with the highest coefficient value of 0.413 (b = 0.413, t = 12.881, ρ < 0.01). Moreover, user intention is significantly impacted by gender and place of living but not by education or age. The research also provides pre-market insights on users' personal types that represent who will most likely adopt the new smart glasses and that differentiate them based on their priorities. To the best of our knowledge, this is among the first works to investigate technology acceptance drivers of AR smart glasses in the Middle East.

A Practical and Efficient Node Blind SignCryption Scheme for the IoT Device Network

In recent years, Internet of Things (IoT for short) research has become one of the top ten most popular research topics. IoT devices also embed many sensing chips for detecting physical signals from the outside environment. In the wireless sensing network (WSN for short), a human can wear several IoT devices around her/his body such as a smart watch, smart band, smart glasses, etc. These IoT devices can collect analog environment data around the user’s body and store these data into memory after data processing. Thus far, we have discovered that some IoT devices have resource limitations such as power shortages or insufficient memory for data computation and preservation. An IoT device such as a smart band attempts to upload a user’s body information to the cloud server by adopting the public-key crypto-system to generate the corresponding cipher-text and related signature for concrete data security; in this situation, the computation time increases linearly and the device can run out of memory, which is inconvenient for users. For this reason, we consider that, if the smart IoT device can perform encryption and signature simultaneously, it can save significant resources for the execution of other applications. As a result, our approach is to design an efficient, practical, and lightweight, blind sign-cryption (SC for short) scheme for IoT device usage. Not only can our methodology offer the sensed data privacy protection efficiently, but it is also fit for the above application scenario with limited resource conditions such as battery shortage or less memory space in the IoT device network.

A CNN-based Human Activity Recognition System Combining a Laser Feedback Interferometry Eye Movement Sensor and an IMU for Context-aware Smart Glasses

Smart glasses are considered the next breakthrough in wearables. As the successor of smart watches and smart ear wear, they promise to extend reality by immersive embedding of content in the user's field of view. While advancements in display technology seems to fulfill this promises, interaction concepts are derived from established wearable concepts like touch interaction or voice interaction, preventing full immersion as they require the user to frequently interact with the glasses. To minimize interactions, we propose to add context-awareness to smart glasses through human activity recognition (HAR) by combining head- and eye movement features to recognize a wide range of activities. To measure eye movements in unobtrusive way, we propose laser feedback interferometry (LFI) sensors. These tiny low power sensors are highly robust to ambient light. We combine LFI sensors and an IMU to collect eye and head movement features from 15 participants performing 7 cognitive and physical activities, leading to a unique data set. To recognize activities we propose a 1D-CNN model and apply transfer learning to personalize the classification, leading to an outstanding macro-F1 score of 88.15 % which outperforms state of the art methods. Finally, we discuss the applicability of the proposed system in a smart glasses setup.

Tailoring assistive smart glasses according to pathologies of visually impaired individuals: an exploratory investigation on social needs and difficulties experienced by visually impaired individuals

Recent advances in the field of assistive devices technology represent a great opportunity for improving the quality of life of people with moderate to severe visual impairment. However, it is still unclear what are the precise daily difficulties, needs and expectations of the smart glasses technology for visually impaired individuals. To this aim, we conducted a survey based on three questionnaires to provide qualitative and quantitative insights on those questions across five groups suffering from various visual pathologies ($$N=50$$ N = 50 ). The results clearly showed the importance of developing tailored solutions to fulfill the heterogeneous daily difficulties and needs identified across pathologies. Overall, groups shared similar expectations regarding the assistive smart glasses functionalities in order to improve social interactions.

Intelligent Digital Twins and Augmented Reality in Inspection and Maintenance

Objective/Scope Simplify and semi-automate the creation of Intelligent Digital Twins available on an augmented hybrid solution (on/offline) based on the latest mobile solutions and wearable technologies (Smart Glasses) communicating over Bluetooth. The main objective was to implement an interactive environment of digital twins through Augmented Reality (AR) to improve decision-making, enhance performance and optimize efficiencies in Inspection and Maintenance. Methods, Procedures, Process A breakthrough algorithm enabled to semi-automate the creation of 3D models by providing a platform of Intelligent Digital Twins to connect to various disparate systems and use Artificial Intelligence-driven augmentation techniques to contextualize and enrich the model with Tag numbers from the asset register. An application has been developed for Smart Glasses by transforming the Digital Twins into sectioned files to enable a performant AR experience and an on/offline syncing functionality with the mobile solution over Bluetooth. Furthermore, the 3D model was embedded within the mobile application for visual support during Inspection and Maintenance. Results, Observations, Conclusions As a result, the following products have been developed to improve efficiency and productivity by leveraging innovative technologies: Application for Intelligent Digital Twins: Single Source of Truth for managing and maintaining Intelligent Digital Twins using breakthrough technology to semi-automate the creation of 3D models from Point Cloud data. A Mark-Up Tool and a Contextualization Editor were developed on the promise of Digital Twins to introduce a communication, collaboration, and planning tool. Furthermore, the 3D model has been embedded within the mobile application environment for visual support on the field tablets. Application for Smart Glasses (AR Headset): An application has been developed for the latest AR Headset to enable an Augmented Reality experience and a Mark-Up tool with Bluetooth communication to the mobile application as well as to support a hybrid on/offline syncing functionality as well as data capturing on the fly. With its basic functionality of hands-free video streaming and communication functionality, the AR Headset enabled a Remote Pre-Commissioning, Certification, and Audit with vendors and independent certifiers during the pandemic. Application Programming Interfaces (API): API interfaces were established to various source systems like the Computerized Maintenance Management System, Inspection Data Management System, Drawing System, and the Process Information System to integrate and visualize the big data within the platform of Intelligent Digital Twins. Novel/Additive Information The use of automation and artificial intelligence (AI) driven technologies for creating Intelligent Digital Twins is a key enabler for global scalability and accelerates creating the company's digital backbone during the digital transformation. Enhancing this technology with an Augmented Reality experience, including mark-up annotations, provides a solid basis for data insights, data-driven decision making, and performance optimization.