Virtual Reality Aided High-Quality 3D Reconstruction by Remote Drones

Artificial intelligence including deep learning and 3D reconstruction methods is changing the daily life of people. Now, an unmanned aerial vehicle that can move freely in the air and avoid harsh ground conditions has been commonly adopted as a suitable tool for 3D reconstruction. The traditional 3D reconstruction mission based on drones usually consists of two steps: image collection and offline post-processing. But there are two problems: one is the uncertainty of whether all parts of the target object are covered, and another is the tedious post-processing time. Inspired by modern deep learning methods, we build a telexistence drone system with an onboard deep learning computation module and a wireless data transmission module that perform incremental real-time dense reconstruction of urban cities by itself. Two technical contributions are proposed to solve the preceding issues. First, based on the popular depth fusion surface reconstruction framework, we combine it with a visual-inertial odometry estimator that integrates the inertial measurement unit and allows for robust camera tracking as well as high-accuracy online 3D scan. Second, the capability of real-time 3D reconstruction enables a new rendering technique that can visualize the reconstructed geometry of the target as navigation guidance in the HMD. Therefore, it turns the traditional path-planning-based modeling process into an interactive one, leading to a higher level of scan completeness. The experiments in the simulation system and our real prototype demonstrate an improved quality of the 3D model using our artificial intelligence leveraged drone system.

Interactive virtual 3D image reconstruction to assist renal surgery in patients with fusion anomalies of the kidney

Objective: Renal fusion anomalies are rare and usually present as horseshoe kidneys or crossed fusion ectopia. The complex renal anatomy seen in patients with these anomalies can present a challenge. Pre-operative planning is therefore paramount in the surgical management of these cases. Herein we report the use of interactive virtual three-dimensional (3D) reconstruction to aid renal surgery in patients with fusion anomalies of the kidney. Materials and Methods: A total of seven cases were performed between May 2016 and October 2020. 3D reconstruction was rendered by Innersight Labs using pre-operative computed tomography (CT) scans. Results: Five patients had malignant disease and two patients had benign pathology. Robotic and open operations were performed in four and three patients, respectively. Conclusion: The use of 3D reconstruction in the cases reported in this series allowed for the identification of variations in renal vasculature, and this informed the choice of operative approach. Oxford Centre for Evidence-Based Medicine Evidence Level: 4

3D Reconstruction with Coronary Artery Based on Curve Descriptor and Projection Geometry-Constrained Vasculature Matching

This paper presents a novel method based on a curve descriptor and projection geometry constrained for vessel matching. First, an LM (Leveberg–Marquardt) algorithm is proposed to optimize the matrix of geometric transformation. Combining with parameter adjusting and the trust region method, the error between 3D reconstructed vessel projection and the actual vessel can be minimized. Then, CBOCD (curvature and brightness order curve descriptor) is proposed to indicate the degree of the self-occlusion of blood vessels during angiography. Next, the error matrix constructed from the error of epipolar matching is used in point pairs matching of the vascular through dynamic programming. Finally, the recorded radius of vessels helps to construct ellipse cross-sections and samples on it to get a point set around the centerline and the point set is converted to mesh for reconstructing the surface of vessels. The validity and applicability of the proposed methods have been verified through experiments that result in the significant improvement of 3D reconstruction accuracy in terms of average back-projection errors. Simultaneously, due to precise point-pair matching, the smoothness of the reconstructed 3D coronary artery is guaranteed.

Sym3DNet: Symmetric 3D Prior Network for Single-View 3D Reconstruction

The three-dimensional (3D) symmetry shape plays a critical role in the reconstruction and recognition of 3D objects under occlusion or partial viewpoint observation. Symmetry structure prior is particularly useful in recovering missing or unseen parts of an object. In this work, we propose Sym3DNet for single-view 3D reconstruction, which employs a three-dimensional reflection symmetry structure prior of an object. More specifically, Sym3DNet includes 2D-to-3D encoder-decoder networks followed by a symmetry fusion step and multi-level perceptual loss. The symmetry fusion step builds flipped and overlapped 3D shapes that are fed to a 3D shape encoder to calculate the multi-level perceptual loss. Perceptual loss calculated in different feature spaces counts on not only voxel-wise shape symmetry but also on the overall global symmetry shape of an object. Experimental evaluations are conducted on both large-scale synthetic 3D data (ShapeNet) and real-world 3D data (Pix3D). The proposed method outperforms state-of-the-art approaches in terms of efficiency and accuracy on both synthetic and real-world datasets. To demonstrate the generalization ability of our approach, we conduct an experiment with unseen category samples of ShapeNet, exhibiting promising reconstruction results as well.

Multi-Scale Presentation of Spatial Context for Cultural Heritage Applications

An approach to the representation and presentation of spatial and geographical context of cultural heritage sites is proposed. The goal is to combine semantic representations of social and historical context with 3D representations of cultural heritage sites acquired through 3D reconstruction and 3D modeling technologies, to support their interpretation and presentation in education and tourism. Several use cases support and demonstrate the application of the proposed approach including immersive craft and context demonstration environment and interactive games.

Human Remains Identification Using Micro-CT, Spectroscopic and A.I. Methods in Forensic Experimental Reconstruction of Dental Patterns After Concentrated Acid Significant Impact

(1) Human teeth are the most resilient tissues in the body. However, exposure to concentrated acids might lead to their obliteration, thus making human identification difficult. Teeth often contain dental restorations from materials that are even more resilient to acid impact. This paper introduces novel method of 3D reconstruction of dental patterns as a crucial step for digital identification with dental records.; (2) With combination of modern methods of Micro-Computed Tomography, Cone Beam Computed Tomography, Attenuated Total Reflection in conjunction with Fourier-Transform Infrared Spectroscopy and Artificial Intelligence Convolutional Neural Network algorithms, the paper presents the way of 3D dental pattern reconstruction and human remains identification. Research studies morphology of teeth, bone and dental materials (Amalgam, Composite, Glass-ionomer cement) under different periods of exposure to 75% sulfuric acid; (3) Results reveal significant volume loss in bone, enamel, dentine and as well glass-ionomer cement. Results also reveal significant resistance of composite and amalgam dental materials to sulfuric acid impact, thus serving as strong parts in the dental pattern mosaic. Paper also introduces probably first successful artificial intelligence application in automated forensic CBCT segmentation.; (4) Interdisciplinary cooperation utilizing mentioned technologies can solve problem of human remains identification with 3D reconstruction of dental patterns and their 2D projections over existing ante-mortem records.

SSOP Three-Dimensional Reconstruction of Tibia and Fibula for Applications in Biomechanical Fracture Models

Context: Non-fatal injuries represent a public health issue. Among them, lower limb fractures have a large impact on the costs related to orthopedic treatments. In this work, a three-dimensional reconstruction of the tibia and fibula was performed for biomechanical applications with the purpose of defining the 3D reconstruction parameters that allow reducing patients’ radiation exposure and computational costs. Method: For the 3D reconstruction, a computerized tomography taken from a volunteer was used, as well as two software applications specialized in DICOM image reconstruction (Mimics Research and 3DSlicer). The number of images included in the volume was modified, and the results were compared. The quality of the reconstructed volumes was verified by comparing the reference volume reconstructed with the total number of images/slices vs. the modified volumes. The MeshLab software was used for this purpose. The analyzed parameters were the distance differences between the reference and the alternative models, as well as the qualitative curvature analysis. Results: The ANOVA results for the Max (maximum distance between meshes) response shows that software and slices are significant factors. However, the software-slices interaction did not have a significant influence. As for the RMS (root mean square) distance response, software, slices, and the software-slices interaction are not significant. For the Mean distance response, slices and the software-slices interaction are not significant. Nevertheless, software significantly influences the response. These results suggest a potential way to reduce the computational cost and the patient’s radiation exposure in future biomechanical and preoperatory analyses, since the same quality can be obtained by including fewer 2D images in the reconstruction. Conclusions: The reconstructed surfaces are smoother when Mimics is used, even though the same smoothness factor was employed in both software applications during the reconstruction. When 16 slices are used (retained every 16 images from the complete original model), the distance differences increased for both bones (tibia and fibula). For the RMS, reducing the number of slices and using either one of the two applications analyzed would not show any significant differences in the reconstruction, thus allowing the potential reduction of radiation exposure of the patient. Acknowledgements: The authors are grateful to Universidad Nacional de Colombia for funding the project “Estado de esfuerzos en un elemento de osteosíntesis en la consolidación de una fractura de miembro inferior”.