Creating and Augmenting Keyboards for Extended Reality with the K eyboard A ugmentation T oolkit

This article discusses the Keyboard Augmentation Toolkit (KAT), which supports the creation of virtual keyboards that can be used both for standalone input (e.g., for mid-air text entry) and to augment physically tracked keyboards/surfaces in mixed reality. In a user study, we firstly examine the impact and pitfalls of visualising shortcuts on a tracked physical keyboard, exploring the utility of virtual per-keycap displays. Supported by this and other recent developments in XR keyboard research, we then describe the design, development, and evaluation-by-demonstration of KAT. KAT simplifies the creation of virtual keyboards (optionally bound to a tracked physical keyboard) that support enhanced display —2D/3D per-key content that conforms to the virtual key bounds; enhanced interactivity —supporting extensible per-key states such as tap, dwell, touch, swipe; flexible keyboard mappings that can encapsulate groups of interaction and display elements, e.g., enabling application-dependent interactions; and flexible layouts —allowing the virtual keyboard to merge with and augment a physical keyboard, or switch to an alternate layout (e.g., mid-air) based on need. Through these features, KAT will assist researchers in the prototyping, creation and replication of XR keyboard experiences, fundamentally altering the keyboard’s form and function.

Augmented, Virtual and Mixed Reality in Dentistry: A Narrative Review on the Existing Platforms and Future Challenges

The recent advancements in digital technologies have led to exponential progress in dentistry. This narrative review aims to summarize the applications of Augmented Reality, Virtual Reality and Mixed Reality in dentistry and describes future challenges in digitalization, such as Artificial Intelligence and Robotics. Augmented Reality, Virtual Reality and Mixed Reality represent effective tools in the educational technology, as they can enhance students’ learning and clinical training. Augmented Reality and Virtual Reality and can also be useful aids during clinical practice. Augmented Reality can be used to add digital data to real life clinical data. Clinicians can apply Virtual Reality for a digital wax-up that provides a pre-visualization of the final post treatment result. In addition, both these technologies may also be employed to eradicate dental phobia in patients and further enhance patient’s education. Similarly, they can be used to enhance communication between the dentist, patient, and technician. Artificial Intelligence and Robotics can also improve clinical practice. Artificial Intelligence is currently developed to improve dental diagnosis and provide more precise prognoses of dental diseases, whereas Robotics may be used to assist in daily practice.

A Human-Centric Model for Sustainable Asset Management in Railway: A Case Study

The ongoing digital transformation is changing asset management in the railway industry. Emerging digital technologies and Artificial Intelligence is expected to facilitate decision-making in management, operation, and maintenance of railway by providing an integrated data-driven and model-driven solution. An important aspect when developing decision-support solutions based on AI and digital technology is the users’ experience. User experience design process aims to create relevance, context-awareness, and meaningfulness for the end-user. In railway contexts, it is believed that applying a human-centric design model in the development of AI-based artefacts, will enhance the usability of the solution, which will have a positive impact on the decision-making processes. In this research, the applicability of such advanced technologies i.e., Virtual Reality, Mixed Reality, and AI have been reviewed for the railway asset management. To carry out this research work, literature review has been conducted related to available Virtual Reality/Augmented Reality/Mixed Reality technologies and their applications within railway industry. It has been found that these technologies are available, but not applied in railway asset management. Thus, the aim of this paper is to propose a human-centric design model for the enhancement of railway asset management using Artificial Intelligence, Virtual Reality, and Mixed Reality technologies. The practical implication of the findings from this work will benefit in increased efficiency and effectiveness of the operation and maintenance processes in railway.

Accuracy of Mixed Reality Combined With Surgical Navigation Assisted Oral and Maxillofacial Tumor Resection

ObjectiveTo evaluate the feasibility and accuracy of mixed reality combined with surgical navigation in oral and maxillofacial tumor surgery.MethodsRetrospective analysis of data of seven patients with oral and maxillofacial tumors who underwent surgery between January 2019 and January 2021 using a combination of mixed reality and surgical navigation. Virtual surgical planning and navigation plan were based on preoperative CT datasets. Through IGT-Link port, mixed reality workstation was synchronized with surgical navigation, and surgical planning data were transferred to the mixed reality workstation. Osteotomy lines were marked with the aid of both surgical navigation and mixed reality images visualized through HoloLens. Frozen section examination was used to ensure negative surgical margins. Postoperative CT datasets were obtained 1 week after the surgery, and chromatographic analysis of virtual osteotomies and actual osteotomies was carried out. Patients received standard oncological postoperative follow-up.ResultsOf the seven patients, four had maxillary tumors and three had mandibular tumors. There were total of 13 osteotomy planes. Mean deviation between the planned osteotomy plane and the actual osteotomy plane was 1.68 ± 0.92 mm; the maximum deviation was 3.46 mm. Chromatographic analysis showed error of ≤3 mm for 80.16% of the points. Mean deviations of maxillary and mandibular osteotomy lines were approximate (1.60 ± 0.93 mm vs. 1.86 ± 0.93 mm). While five patients had benign tumors, two had malignant tumors. Mean deviations of osteotomy lines was comparable between patients with benign and malignant tumors (1.48 ± 0.74 mm vs. 2.18 ± 0.77 mm). Intraoperative frozen pathology confirmed negative resection margins in all cases. No tumor recurrence or complications occurred during mean follow-up of 15.7 months (range, 6-26 months).ConclusionThe combination of mixed reality technology and surgical navigation appears to be feasible, safe, and effective for tumor resection in the oral and maxillofacial region.

Efficient Ray Tracing of Large 3D Scenes for Mobile Distributed Computing Environments

Cluster computing has attracted much attention as an effective way of solving large-scale problems. However, only a few attempts have been made to explore mobile computing clusters that can be easily built using commodity smartphones and tablets. To investigate the possibility of mobile cluster-based rendering of large datasets, we developed a mobile GPU ray tracer that renders nontrivial 3D scenes with many millions of triangles at an interactive frame rate on a small-scale mobile cluster. To cope with the limited processing power and memory space, we first present an effective 3D scene representation scheme suitable for mobile GPU rendering. Then, to avoid performance impairment caused by the high latency and low bandwidth of mobile networks, we propose using a static load balancing strategy, which we found to be more appropriate for the vulnerable mobile clustering environment than a dynamic strategy. Our mobile distributed rendering system achieved a few frames per second when ray tracing 1024 × 1024 images, using only 16 low-end smartphones, for large 3D scenes, some with more than 10 million triangles. Through a conceptual demonstration, we also show that the presented rendering scheme can be effectively explored for augmenting real scene images, captured or perceived by augmented and mixed reality devices, with high quality ray-traced images.