scholarly journals Segmentation of Illuminated Areas of Light Using CNN and Large-Scale RGB+D Dataset for Augmented and Mixed Reality Systems

2020 ◽  
pp. short58-1-short58-7
Author(s):  
Maksim Sorokin ◽  
Dmitriy Zhdanov ◽  
Andrey Zhdanov
Keyword(s):  
Neural Networks ◽  
Real Time ◽  
Convolutional Neural Networks ◽  
Large Scale ◽  
Mixed Reality ◽  
Depth Map ◽  
Light Sources ◽  
Realistic Rendering ◽  
Visual Difference

This work is devoted to the problem of restoring realistic rendering for augmented and mixed reality systems. Finding the light sources and restoring the correct distribution of scene brightness is one of the key parameters that allows to solve the problem of correct interaction between the virtual and real worlds. With the advent of such datasets as, "LARGE-SCALE RGB + D," it became possible to train neural networks to recognize the depth map of images, which is a key requirement for working with the environment in real time. Additionally, in this work, convolutional neural networks were trained on the synthesized dataset with realistic lighting. The results of the proposed methods are presented, the accuracy of restoring the position of the light sources is estimated, and the visual difference between the image of the scene with the original light sources and the same scene. The speed allows it to be used in real-time AR/VR systems.

scholarly journals The efficient method of illumination restoration in mixed reality systems with use of HDR image of 3D scene

2019 ◽  
Author(s):  
Николай Богданов ◽  
Nikolay Bogdanov ◽  
Игорь Потемин ◽  
Igor' Potemin ◽  
Дмитрий Жданов ◽  
...  
Keyword(s):  
Neural Networks ◽  
Mixed Reality ◽  
Brightness Distribution ◽  
Line Of Sight ◽  
Light Sources ◽  
Virtual Objects ◽  
Correct Representation ◽  
3D Scene ◽  
Visual Difference ◽  
Hdr Image

One of the main problems of mixed reality devices is the physically correct representation of the brightness distribution for virtual objects and their shadows in the real world. In other words, restoring the correct distribution of scene brightness is one of the key parameters to solve the problem of correct interaction between the virtual and real worlds, but neural networks do not allow to determine the position of light sources that are not in line of sight. The paper proposes a method for restoring the parameters of light sources based on the analysis of shadows cast by objects. The results of the proposed method are presented, the accuracy of restoring the position of light sources is estimated and the visual difference between the image of the scene with the original light sources from the same scene with the restored parameters of light sources is demonstrated.

Real-time Detection of Aortic Valve in Echocardiography using Convolutional Neural Networks

2020 ◽  
Vol 16 (5) ◽  
pp. 584-591 ◽  
Author(s):  
Muhammad Hanif Ahmad Nizar ◽  
Chow Khuen Chan ◽  
Azira Khalil ◽  
Ahmad Khairuddin Mohamed Yusof ◽  
Khin Wee Lai
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Heart Disease ◽  
Aortic Valve ◽  
Real Time ◽  
Convolutional Neural Networks ◽  
Valvular Heart Disease ◽  
Detection System ◽  
Processing Unit ◽  
Real Time Detection

Background: Valvular heart disease is a serious disease leading to mortality and increasing medical care cost. The aortic valve is the most common valve affected by this disease. Doctors rely on echocardiogram for diagnosing and evaluating valvular heart disease. However, the images from echocardiogram are poor in comparison to Computerized Tomography and Magnetic Resonance Imaging scan. This study proposes the development of Convolutional Neural Networks (CNN) that can function optimally during a live echocardiographic examination for detection of the aortic valve. An automated detection system in an echocardiogram will improve the accuracy of medical diagnosis and can provide further medical analysis from the resulting detection. Methods: Two detection architectures, Single Shot Multibox Detector (SSD) and Faster Regional based Convolutional Neural Network (R-CNN) with various feature extractors were trained on echocardiography images from 33 patients. Thereafter, the models were tested on 10 echocardiography videos. Results: Faster R-CNN Inception v2 had shown the highest accuracy (98.6%) followed closely by SSD Mobilenet v2. In terms of speed, SSD Mobilenet v2 resulted in a loss of 46.81% in framesper- second (fps) during real-time detection but managed to perform better than the other neural network models. Additionally, SSD Mobilenet v2 used the least amount of Graphic Processing Unit (GPU) but the Central Processing Unit (CPU) usage was relatively similar throughout all models. Conclusion: Our findings provide a foundation for implementing a convolutional detection system to echocardiography for medical purposes.

scholarly journals Real-time intraoperative glioma diagnosis using fluorescence imaging and deep convolutional neural networks

2021 ◽  
Author(s):  
Biluo Shen ◽  
Zhe Zhang ◽  
Xiaojing Shi ◽  
Caiguang Cao ◽  
Zeyu Zhang ◽  
...  
Keyword(s):  
Neural Networks ◽  
Real Time ◽  
Fluorescence Imaging ◽  
Convolutional Neural Networks ◽  
Frozen Section ◽  
Fluorescent Imaging ◽  
Pathological Diagnosis ◽  
Pathological Examination ◽  
Deep Convolutional Neural Networks ◽  
Fluorescence Images

Abstract Purpose Surgery is the predominant treatment modality of human glioma but suffers difficulty on clearly identifying tumor boundaries in clinic. Conventional practice involves neurosurgeon’s visual evaluation and intraoperative histological examination of dissected tissues using frozen section, which is time-consuming and complex. The aim of this study was to develop fluorescent imaging coupled with artificial intelligence technique to quickly and accurately determine glioma in real-time during surgery. Methods Glioma patients (N = 23) were enrolled and injected with indocyanine green for fluorescence image–guided surgery. Tissue samples (N = 1874) were harvested from surgery of these patients, and the second near-infrared window (NIR-II, 1000–1700 nm) fluorescence images were obtained. Deep convolutional neural networks (CNNs) combined with NIR-II fluorescence imaging (named as FL-CNN) were explored to automatically provide pathological diagnosis of glioma in situ in real-time during patient surgery. The pathological examination results were used as the gold standard. Results The developed FL-CNN achieved the area under the curve (AUC) of 0.945. Comparing to neurosurgeons’ judgment, with the same level of specificity >80%, FL-CNN achieved a much higher sensitivity (93.8% versus 82.0%, P < 0.001) with zero time overhead. Further experiments demonstrated that FL-CNN corrected >70% of the errors made by neurosurgeons. FL-CNN was also able to rapidly predict grade and Ki-67 level (AUC 0.810 and 0.625) of tumor specimens intraoperatively. Conclusion Our study demonstrates that deep CNNs are better at capturing important information from fluorescence images than surgeons’ evaluation during patient surgery. FL-CNN is highly promising to provide pathological diagnosis intraoperatively and assist neurosurgeons to obtain maximum resection safely. Trial registration ChiCTR ChiCTR2000029402. Registered 29 January 2020, retrospectively registered

Sign in / Sign up

Export Citation Format

Share Document