Share-Z: Client/Server Depth Sensing for See-Through Head-Mounted Displays

2002 ◽  
Vol 11 (2) ◽  
pp. 176-188 ◽  
Author(s):  
Yuichi Ohta ◽  
Yasuyuki Sugaya ◽  
Hiroki Igarashi ◽  
Toshikazu Ohtsuki ◽  
Kaito Taguchi
Keyword(s):  
Real World ◽  
Mixed Reality ◽  
Depth Map ◽  
Depth Information ◽  
Depth Sensor ◽  
Depth Sensing ◽  
Client Server ◽  
The Real ◽  
Depth Sensors ◽  
Viewing Position

In mixed reality, occlusions and shadows are important to realize a natural fusion between the real and virtual worlds. In order to achieve this, it is necessary to acquire dense depth information of the real world from the observer's viewing position. The depth sensor must be attached to the see-through HMD of the observer because he/she moves around. The sensor should be small and light enough to be attached to the HMD and should be able to produce a reliable dense depth map at video rate. Unfortunately, however, no such depth sensors are available. We propose a client/server depth-sensing scheme to solve this problem. A server sensor located at a fixed position in the real world acquires the 3-D information of the world, and a client sensor attached to each observer produces the depth map from his/her viewing position using the 3-D information supplied from the server. Multiple clients can share the 3-D information of the server; we call it Share-Z. In this paper, the concept and merits of Share-Z are discussed. An experimental system developed to demonstrate the feasibility of Share-Z is also described.

scholarly journals GEUINF: Real-Time Visualization of Indoor Facilities Using Mixed Reality

Sensors ◽  
2021 ◽  
Vol 21 (4) ◽  
pp. 1123
Author(s):  
David Jurado ◽  
Juan M. Jurado ◽  
Lidia Ortega ◽  
Francisco R. Feito
Keyword(s):  
Real Time ◽  
Real World ◽  
Mixed Reality ◽  
3D Models ◽  
Indoor Navigation ◽  
Depth Sensing ◽  
3D Vision ◽  
The Real ◽  
3D Data ◽  
Real Time Visualization

Mixed reality (MR) enables a novel way to visualize virtual objects on real scenarios considering physical constraints. This technology arises with other significant advances in the field of sensors fusion for human-centric 3D capturing. Recent advances for scanning the user environment, real-time visualization and 3D vision using ubiquitous systems like smartphones allow us to capture 3D data from the real world. In this paper, a disruptive application for assessing the status of indoor infrastructures is proposed. The installation and maintenance of hidden facilities such as water pipes, electrical lines and air conditioning tubes, which are usually occluded behind the wall, supposes tedious and inefficient tasks. Most of these infrastructures are digitized but they cannot be visualized onsite. In this research, we focused on the development of a new application (GEUINF) to be launched on smartphones that are capable of capturing 3D data of the real world by depth sensing. This information is relevant to determine the user position and orientation. Although previous approaches used fixed markers for this purpose, our application enables the estimation of both parameters with a centimeter accuracy without them. This novelty is possible since our method is based on a matching process between reconstructed walls of the real world and 3D planes of the replicated world in a virtual environment. Our markerless approach is based on scanning planar surfaces of the user environment and then, these are geometrically aligned with their corresponding virtual 3D entities. In a preprocessing phase, the 2D CAD geometry available from an architectural project is used to generate 3D models of an indoor building structure. In real time, these virtual elements are tracked with the real ones modeled by using ARCore library. Once the alignment between virtual and real worlds is done, the application enables the visualization, navigation and interaction with the virtual facility networks in real-time. Thus, our method may be used by private companies and public institutions responsible of the indoor facilities management and also may be integrated with other applications focused on indoor navigation.

Real-world environment affects the color appearance of virtual stimuli produced by augmented reality

2019 ◽  
Vol 2019 (1) ◽  
pp. 237-242
Author(s):  
Siyuan Chen ◽  
Minchen Wei
Keyword(s):  
Augmented Reality ◽  
Real World ◽  
Mixed Reality ◽  
Rapid Development ◽  
Light Level ◽  
Color Appearance ◽  
Chromatic Adaptation ◽  
Lighting Condition ◽  
The Real ◽  
World Environment

Color appearance models have been extensively studied for characterizing and predicting the perceived color appearance of physical color stimuli under different viewing conditions. These stimuli are either surface colors reflecting illumination or self-luminous emitting radiations. With the rapid development of augmented reality (AR) and mixed reality (MR), it is critically important to understand how the color appearance of the objects that are produced by AR and MR are perceived, especially when these objects are overlaid on the real world. In this study, nine lighting conditions, with different correlated color temperature (CCT) levels and light levels, were created in a real-world environment. Under each lighting condition, human observers adjusted the color appearance of a virtual stimulus, which was overlaid on a real-world luminous environment, until it appeared the whitest. It was found that the CCT and light level of the real-world environment significantly affected the color appearance of the white stimulus, especially when the light level was high. Moreover, a lower degree of chromatic adaptation was found for viewing the virtual stimulus that was overlaid on the real world.

scholarly journals BLAINDER—A Blender AI Add-On for Generation of Semantically Labeled Depth-Sensing Data

Sensors ◽  
2021 ◽  
Vol 21 (6) ◽  
pp. 2144
Author(s):  
Stefan Reitmann ◽  
Lorenzo Neumann ◽  
Bernhard Jung
Keyword(s):  
Point Clouds ◽  
Training Data ◽  
Sensor Data ◽  
Generation Process ◽  
Data Generation ◽  
Depth Sensor ◽  
Depth Sensing ◽  
Depth Sensors ◽  
3D Point Clouds ◽  
Wide Range

Common Machine-Learning (ML) approaches for scene classification require a large amount of training data. However, for classification of depth sensor data, in contrast to image data, relatively few databases are publicly available and manual generation of semantically labeled 3D point clouds is an even more time-consuming task. To simplify the training data generation process for a wide range of domains, we have developed the BLAINDER add-on package for the open-source 3D modeling software Blender, which enables a largely automated generation of semantically annotated point-cloud data in virtual 3D environments. In this paper, we focus on classical depth-sensing techniques Light Detection and Ranging (LiDAR) and Sound Navigation and Ranging (Sonar). Within the BLAINDER add-on, different depth sensors can be loaded from presets, customized sensors can be implemented and different environmental conditions (e.g., influence of rain, dust) can be simulated. The semantically labeled data can be exported to various 2D and 3D formats and are thus optimized for different ML applications and visualizations. In addition, semantically labeled images can be exported using the rendering functionalities of Blender.

scholarly journals Metazoa Ludens: Mixed Reality Environment for Playing Computer Games with Pets

2006 ◽  
Vol 5 (3) ◽  
pp. 53-58 ◽  
Author(s):  
Roger K. C. Tan ◽  
Adrian David Cheok ◽  
James K. S. Teh
Keyword(s):  
Real World ◽  
Computer Games ◽  
Virtual World ◽  
Mixed Reality ◽  
Leisure Activities ◽  
Computer Game ◽  
Technological Advancement ◽  
The Real ◽  
Computer Mediated ◽  
Pet Owner

For better or worse, technological advancement has changed the world to the extent that at a professional level demands from the working executive required more hours either in the office or on business trips, on a social level the population (especially the younger generation) are glued to the computer either playing video games or surfing the internet. Traditional leisure activities, especially interaction with pets have been neglected or forgotten. This paper introduces Metazoa Ludens, a new computer mediated gaming system which allows pets to play new mixed reality computer games with humans via custom built technologies and applications. During the game-play the real pet chases after a physical movable bait in the real world within a predefined area; infra-red camera tracks the pets' movements and translates them into the virtual world of the system, corresponding them to the movement of a virtual pet avatar running after a virtual human avatar. The human player plays the game by controlling the human avatar's movements in the virtual world, this in turn relates to the movements of the physical movable bait in the real world which moves as the human avatar does. This unique way of playing computer game would give rise to a whole new way of mixed reality interaction between the pet owner and her pet thereby bringing technology and its influence on leisure and social activities to the next level

scholarly journals Approximate Depth Shape Reconstruction for RGB-D Images Captured from HMDs for Mixed Reality Applications

2020 ◽  
Vol 6 (3) ◽  
pp. 11
Author(s):  
Naoyuki Awano
Keyword(s):  
Mixed Reality ◽  
Signal To Noise Ratio ◽  
Low Cost ◽  
Real Space ◽  
Shape Reconstruction ◽  
Human Vision ◽  
Depth Image ◽  
Depth Sensor ◽  
Depth Sensors ◽  
Object Shapes

Depth sensors are important in several fields to recognize real space. However, there are cases where most depth values in a depth image captured by a sensor are constrained because the depths of distal objects are not always captured. This often occurs when a low-cost depth sensor or structured-light depth sensor is used. This also occurs frequently in applications where depth sensors are used to replicate human vision, e.g., when using the sensors in head-mounted displays (HMDs). One ideal inpainting (repair or restoration) approach for depth images with large missing areas, such as partial foreground depths, is to inpaint only the foreground; however, conventional inpainting studies have attempted to inpaint entire images. Thus, under the assumption of an HMD-mounted depth sensor, we propose a method to inpaint partially and reconstruct an RGB-D depth image to preserve foreground shapes. The proposed method is comprised of a smoothing process for noise reduction, filling defects in the foreground area, and refining the filled depths. Experimental results demonstrate that the inpainted results produced using the proposed method preserve object shapes in the foreground area with accurate results of the inpainted area with respect to the real depth with the peak signal-to-noise ratio metric.

scholarly journals CrossFuNet: RGB and Depth Cross-Fusion Network for Hand Pose Estimation

Sensors ◽  
2021 ◽  
Vol 21 (18) ◽  
pp. 6095
Author(s):  
Xiaojing Sun ◽  
Bin Wang ◽  
Longxiang Huang ◽  
Qian Zhang ◽  
Sulei Zhu ◽  
...  
Keyword(s):  
Pose Estimation ◽  
Depth Map ◽  
Depth Information ◽  
Feature Maps ◽  
Hand Pose Estimation ◽  
Depth Sensors ◽  
Key Points ◽  
Rgb Images ◽  
Public Datasets ◽  
Hand Pose

Despite recent successes in hand pose estimation from RGB images or depth maps, inherent challenges remain. RGB-based methods suffer from heavy self-occlusions and depth ambiguity. Depth sensors rely heavily on distance and can only be used indoors, thus there are many limitations to the practical application of depth-based methods. The aforementioned challenges have inspired us to combine the two modalities to offset the shortcomings of the other. In this paper, we propose a novel RGB and depth information fusion network to improve the accuracy of 3D hand pose estimation, which is called CrossFuNet. Specifically, the RGB image and the paired depth map are input into two different subnetworks, respectively. The feature maps are fused in the fusion module in which we propose a completely new approach to combine the information from the two modalities. Then, the common method is used to regress the 3D key-points by heatmaps. We validate our model on two public datasets and the results reveal that our model outperforms the state-of-the-art methods.

scholarly journals Challenges in Surgical Training- Exploring the role of virtual and augmented reality

2020 ◽  
Vol 3 (1) ◽  
pp. 9-10
Author(s):  
Rehan Ahmed Khan
Keyword(s):  
Virtual Reality ◽  
Augmented Reality ◽  
Surgical Training ◽  
Real World ◽  
Mixed Reality ◽  
Virtual Patients ◽  
Virtual Image ◽  
The Real ◽  
Surgical Residents ◽  
Number Of Patients

In the field of surgery, major changes that have occurred include the advent of minimally invasive surgery and the realization of the importance of the ‘systems’ in the surgical care of the patient (Pierorazio & Allaf, 2009). Challenges in surgical training are two-fold: (i) to train the surgical residents to manage a patient clinically (ii) to train them in operative skills (Singh & Darzi,2013). In Pakistan, another issue with surgical training is that we have the shortest duration of surgical training in general surgery of four years only, compared to six to eight years in Europe and America (Zafar & Rana, 2013). Along with it, the smaller number of patients to surgical residents’ ratio is also an issue in surgical training. This warrants formal training outside the operation room. It has been reported by many authors that changes are required in the current surgical training system due to the significant deficiencies in the graduating surgeon (Carlsen et al., 2014; Jarman et al., 2009; Parsons, Blencowe, Hollowood, & Grant, 2011). Considering surgical training, it is imperative that a surgeon is competent in clinical management and operative skills at the end of the surgical training. To achieve this outcome in this challenging scenario, a resident surgeon should be provided with the opportunities of training outside the operation theatre, before s/he can perform procedures on a real patient. The need for this training was felt more when the Institute of Medicine in the USA published a report, ‘To Err is Human’ (Stelfox, Palmisani, Scurlock, Orav, & Bates, 2006), with an aim to reduce medical errors. This is required for better training and objective assessment of the surgical residents. The options for this training include but are not limited to the use of mannequins, virtual patients, virtual simulators, virtual reality, augmented reality, and mixed reality. Simulation is a technique to substitute or add to real experiences with guided ones, often immersive in nature, that reproduce substantial aspects of the real world in a fully interactive way. Mannequins, virtual simulators are in use for a long time now. They are available in low fidelity to high fidelity mannequins and virtual simulators and help residents understand the surgical anatomy, operative site and practice their skills. Virtual patients can be discussed with students in a simple format of the text, pictures, and videos as case files available online, or in the form of customized software applications based on algorithms. In a study done by Courtielle et al, they reported that knowledge retention is increased in residents when it is delivered through virtual patients as compared to lecturing (Courteille et al., 2018).But learning the skills component requires hands-on practice. This gap can be bridged with virtual, augmented, or mixed reality. There are three types of virtual reality (VR) technologies: (i) non-immersive, (ii) semi-immersive, and (iii) fully immersive. Non-immersive (VR) involves the use of software and computers. In semi-immersive and immersive VR, the virtual image is presented through the head-mounted display(HMD), the difference being that in the fully immersive type, the virtual image is completely obscured from the actual world. Using handheld devices with haptic feedback the trainee can perform a procedure in the virtual environment (Douglas, Wilke, Gibson, Petricoin, & Liotta, 2017). Augmented reality (AR) can be divided into complete AR or mixed reality (MR). Through AR and MR, a trainee can see a virtual and a real-world image at the same time, making it easy for the supervisor to explain the steps of the surgery. Similar to VR, in AR and MR the user wears an HMD that shows both images. In AR, the virtual image is transparent whereas, in MR, it appears solid (Douglas et al., 2017). Virtual augmented and mixed reality has more potential to train surgeons as they provide fidelity very close to the real situation and require fewer physical resources and space compared to the simulators. But they are costlier, and affordability is an issue. To overcome this, low-cost solutions to virtual reality have been developed. It is high time that we also start thinking on the same lines and develop this means of training our surgeons at an affordable cost.

scholarly journals Designing an augmented reality video game to assist stroke patients with independent rehabilitation

2021 ◽  
Author(s):  
◽  
Regan Petrie
Keyword(s):  
Augmented Reality ◽  
Video Game ◽  
Real World ◽  
Lower Limb ◽  
Health Care Professionals ◽  
Mixed Reality ◽  
Low Cost ◽  
Stroke Patients ◽  
The Real ◽  
Positive Results

<p>Early, intense practice of functional, repetitive rehabilitation interventions has shown positive results towards lower-limb recovery for stroke patients. However, long-term engagement in daily physical activity is necessary to maximise the physical and cognitive benefits of rehabilitation. The mundane, repetitive nature of traditional physiotherapy interventions and other personal, environmental and physical elements create barriers to participation. It is well documented that stroke patients engage in as little as 30% of their rehabilitation therapies. Digital gamified systems have shown positive results towards addressing these barriers of engagement in rehabilitation, but there is a lack of low-cost commercially available systems that are designed and personalised for home use. At the same time, emerging mixed reality technologies offer the ability to seamlessly integrate digital objects into the real world, generating an immersive, unique virtual world that leverages the physicality of the real world for a personalised, engaging experience.  This thesis explored how the design of an augmented reality exergame can facilitate engagement in independent lower-limb stroke rehabilitation. Our system converted prescribed exercises into active gameplay using commercially available augmented reality mobile technology. Such a system introduced an engaging, interactive alternative to existing mundane physiotherapy exercises.  The development of the system was based on a user-centered iterative design process. The involvement of health care professionals and stroke patients throughout each stage of the design and development process helped understand users’ needs, requirements and environment to refine the system and ensure its validity as a substitute for traditional rehabilitation interventions.  The final output was an augmented reality exergame that progressively facilitates sit-to-stand exercises by offering immersive interactions with digital exotic wildlife. We hypothesize that the immersive, active nature of a mobile, mixed reality exergame will increase engagement in independent task training for lower-limb rehabilitation.</p>

scholarly journals Gap Affordance Judgments in Mixed Reality: Testing the Role of Display Weight and Field of View

2021 ◽  
Vol 2 ◽  
Author(s):  
Holly C. Gagnon ◽  
Yu Zhao ◽  
Matthew Richardson ◽  
Grant D. Pointon ◽  
Jeanine K. Stefanucci ◽  
...  
Keyword(s):  
Real World ◽  
Mixed Reality ◽  
Field Of View ◽  
The Real ◽  
Factors Associated ◽  
Reality Testing ◽  
Action Capabilities ◽  
Two Factors ◽  
Step Over

Measures of perceived affordances—judgments of action capabilities—are an objective way to assess whether users perceive mediated environments similarly to the real world. Previous studies suggest that judgments of stepping over a virtual gap using augmented reality (AR) are underestimated relative to judgments of real-world gaps, which are generally overestimated. Across three experiments, we investigated whether two factors associated with AR devices contributed to the observed underestimation: weight and field of view (FOV). In the first experiment, observers judged whether they could step over virtual gaps while wearing the HoloLens (virtual gaps) or not (real-world gaps). The second experiment tested whether weight contributes to underestimation of perceived affordances by having participants wear the HoloLens during judgments of both virtual and real gaps. We replicated the effect of underestimation of step capabilities in AR as compared to the real world in both Experiments 1 and 2. The third experiment tested whether FOV influenced judgments by simulating a narrow (similar to the HoloLens) FOV in virtual reality (VR). Judgments made with a reduced FOV were compared to judgments made with the wider FOV of the HTC Vive Pro. The results showed relative underestimation of judgments of stepping over gaps in narrow vs. wide FOV VR. Taken together, the results suggest that there is little influence of weight of the HoloLens on perceived affordances for stepping, but that the reduced FOV of the HoloLens may contribute to the underestimation of stepping affordances observed in AR.

scholarly journals Construction of All-in-Focus Images Assisted by Depth Sensing

Sensors ◽  
2019 ◽  
Vol 19 (6) ◽  
pp. 1409 ◽  
Author(s):  
Hang Liu ◽  
Hengyu Li ◽  
Jun Luo ◽  
Shaorong Xie ◽  
Yu Sun
Keyword(s):  
Image Fusion ◽  
Imaging System ◽  
State Of The Art ◽  
Depth Map ◽  
Depth Of Field ◽  
Depth Sensor ◽  
Depth Sensing ◽  
Focus Image ◽  
Limited Depth ◽  
Image Fusion Method

Multi-focus image fusion is a technique for obtaining an all-in-focus image in which all objects are in focus to extend the limited depth of field (DoF) of an imaging system. Different from traditional RGB-based methods, this paper presents a new multi-focus image fusion method assisted by depth sensing. In this work, a depth sensor is used together with a colour camera to capture images of a scene. A graph-based segmentation algorithm is used to segment the depth map from the depth sensor, and the segmented regions are used to guide a focus algorithm to locate in-focus image blocks from among multi-focus source images to construct the reference all-in-focus image. Five test scenes and six evaluation metrics were used to compare the proposed method and representative state-of-the-art algorithms. Experimental results quantitatively demonstrate that this method outperforms existing methods in both speed and quality (in terms of comprehensive fusion metrics). The generated images can potentially be used as reference all-in-focus images.

Sign in / Sign up

Export Citation Format

Share Document