Virtual reality for 3D histology: multi-scale visualization of organs with interactive feature exploration

Abstract Background Virtual reality (VR) enables data visualization in an immersive and engaging manner, and it can be used for creating ways to explore scientific data. Here, we use VR for visualization of 3D histology data, creating a novel interface for digital pathology to aid cancer research. Methods Our contribution includes 3D modeling of a whole organ and embedded objects of interest, fusing the models with associated quantitative features and full resolution serial section patches, and implementing the virtual reality application. Our VR application is multi-scale in nature, covering two object levels representing different ranges of detail, namely organ level and sub-organ level. In addition, the application includes several data layers, including the measured histology image layer and multiple representations of quantitative features computed from the histology. Results In our interactive VR application, the user can set visualization properties, select different samples and features, and interact with various objects, which is not possible in the traditional 2D-image view used in digital pathology. In this work, we used whole mouse prostates (organ level) with prostate cancer tumors (sub-organ objects of interest) as example cases, and included quantitative histological features relevant for tumor biology in the VR model. Conclusions Our application enables a novel way for exploration of high-resolution, multidimensional data for biomedical research purposes, and can also be used in teaching and researcher training. Due to automated processing of the histology data, our application can be easily adopted to visualize other organs and pathologies from various origins.

Download Full-text

A deep-learning-based prognostic nomogram integrating microscopic digital pathology and macroscopic magnetic resonance images in nasopharyngeal carcinoma: a multi-cohort study

Therapeutic Advances in Medical Oncology ◽

10.1177/1758835920971416 ◽

2020 ◽

Vol 12 ◽

pp. 175883592097141

Author(s):

Fan Zhang ◽

Lian-Zhen Zhong ◽

Xun Zhao ◽

Di Dong ◽

Ji-Jin Yao ◽

...

Keyword(s):

Deep Learning ◽

Magnetic Resonance ◽

Nasopharyngeal Carcinoma ◽

Treatment Failure ◽

Digital Pathology ◽

Imaging Features ◽

Multi Scale ◽

External Test ◽

Whole Slide Images ◽

Independent Cohort

Background: To explore the prognostic value of radiomics-based and digital pathology-based imaging biomarkers from macroscopic magnetic resonance imaging (MRI) and microscopic whole-slide images for patients with nasopharyngeal carcinoma (NPC). Methods: We recruited 220 NPC patients and divided them into training ( n = 132), internal test ( n = 44), and external test ( n = 44) cohorts. The primary endpoint was failure-free survival (FFS). Radiomic features were extracted from pretreatment MRI and selected and integrated into a radiomic signature. The histopathological signature was extracted from whole-slide images of biopsy specimens using an end-to-end deep-learning method. Incorporating two signatures and independent clinical factors, a multi-scale nomogram was constructed. We also tested the correlation between the key imaging features and genetic alternations in an independent cohort of 16 patients (biological test cohort). Results: Both radiomic and histopathologic signatures presented significant associations with treatment failure in the three cohorts (C-index: 0.689–0.779, all p < 0.050). The multi-scale nomogram showed a consistent significant improvement for predicting treatment failure compared with the clinical model in the training (C-index: 0.817 versus 0.730, p < 0.050), internal test (C-index: 0.828 versus 0.602, p < 0.050) and external test (C-index: 0.834 versus 0.679, p < 0.050) cohorts. Furthermore, patients were stratified successfully into two groups with distinguishable prognosis (log-rank p < 0.0010) using our nomogram. We also found that two texture features were related to the genetic alternations of chromatin remodeling pathways in another independent cohort. Conclusion: The multi-scale imaging features showed a complementary value in prognostic prediction and may improve individualized treatment in NPC.

Download Full-text

Full-resolution encoder-decoder networks with multi-scale feature fusion for human pose estimation

Proceedings of the 2nd ACM International Conference on Multimedia in Asia ◽

10.1145/3444685.3446282 ◽

2021 ◽

Author(s):

Jie Ou ◽

Mingjian Chen ◽

Hong Wu

Keyword(s):

Pose Estimation ◽

Feature Fusion ◽

Human Pose Estimation ◽

Scale Feature ◽

Multi Scale ◽

Full Resolution ◽

Human Pose

Download Full-text

HapticWall: An Encountered-Type Two-Dimensional Vertical System for Virtual Reality

Volume 2: Intelligent Transportation/Vehicles; Manufacturing; Mechatronics; Engine/After-Treatment Systems; Soft Actuators/Manipulators; Modeling/Validation; Motion/Vibration Control Applications; Multi-Agent/Networked Systems; Path Planning/Motion Control; Renewable/Smart Energy Systems; Security/Privacy of Cyber-Physical Systems; Sensors/Actuators; Tracking Control Systems; Unmanned Ground/Aerial Vehicles; Vehicle Dynamics, Estimation, Control; Vibration/Control Systems; Vibrations ◽

10.1115/dscc2020-3105 ◽

2020 ◽

Author(s):

Yuwei Li ◽

David Donghyun Kim ◽

Brian Anthony

Keyword(s):

Virtual Reality ◽

Large Scale ◽

Haptic Feedback ◽

Dimensional System ◽

Haptic Interfaces ◽

Two Dimensional ◽

Multi Scale ◽

Physical Interactions ◽

Two Dimensional System ◽

Type Motor

Abstract We present HapticWall, an encountered-type, motor actuated vertical two-dimensional system that enables both small and large scale physical interactions in virtual reality. HapticWall consists of a motor-actuated vertical two-dimensional gantry system that powers the physical proxy for the virtual counterpart. The physical proxy, combined with the HapticWall system, can be used to provide both small and large scale haptic feedbacks for virtual reality in the vertical space. Haptic Wall is capable of providing wall-like haptic feedback and interactions in the vertical space. We created two virtual reality applications to demonstrate the application of the HapticWall system. Preliminary user feedback was collected to evaluate the performance and the limitations of the HapticWall system. The results of our study are presented in this paper. The outcome of this research will provide better understanding of multi-scale haptic interfaces in the vertical space for virtual reality and guide the future development of the HapticWall system.

Download Full-text

MULTISENSORY SCIENTIFIC DATA SENSUALIZATION THROUGH VIRTUAL REALITY TECHNOLOGY

Virtual Reality Software and Technology ◽

10.1142/9789814350938_0013 ◽

1994 ◽

Cited By ~ 2

Author(s):

TETSURO OGI ◽

MICHITAKA HIROSE

Keyword(s):

Virtual Reality ◽

Scientific Data ◽

Virtual Reality Technology

Download Full-text

Different Levels of Complexity for Integrating Textured Extra-terrestrial Elevation Data in Game Engines for Educational Augmented and Virtual Reality Applications

KN - Journal of Cartography and Geographic Information ◽

10.1007/s42489-021-00090-3 ◽

2021 ◽

Author(s):

Claudia Lindner ◽

Annette Ortwein ◽

Kilian Staar ◽

Andreas Rienow

Keyword(s):

Virtual Reality ◽

3D Models ◽

Scientific Data ◽

Educational Settings ◽

Game Development ◽

Levels Of Detail ◽

Landing Sites ◽

Elevation Data ◽

Terrestrial Environments ◽

Game Engines

AbstractElevation and visual data from Chang’E-2, Mars Viking, and MOLA were transformed into 3D models and environments using unity and unreal engine to be implemented in augmented (AR) and virtual reality (VR) applications, respectively. The workflows for the two game development engines and the two purposes overlap, but have significant differences stemming from their intended usage: both are used in educational settings, but while the AR app has to run on basic smartphones that students from all socio-economic backgrounds might have, the VR requires high-end PCs and can therefore make use of respective devices’ potential. Hence, the models for the AR app are reduced to the necessary components and sizes of the highest mountains on Luna and Mars, whereas the VR app contains several models of probe landing sites on Mars, a landscape containing the entire planet at multiple levels of detail and a complex environment. Both applications are enhanced for educational use with annotations and interactive elements. This study focuses on the transfer of scientific data into game development engines for the use in educational settings using the example of scales in extra-terrestrial environments.

Download Full-text

Exploring virtual reality technology and the Oculus Rift for the examination of digital pathology slides

Journal of Pathology Informatics ◽

10.4103/2153-3539.181766 ◽

2016 ◽

Vol 7 (1) ◽

pp. 22 ◽

Cited By ~ 21

Author(s):

Navid Farahani ◽

Robert Post ◽

Jon Duboy ◽

Ishtiaque Ahmed ◽

BrianJ Kolowitz ◽

...

Keyword(s):

Virtual Reality ◽

Digital Pathology ◽

Virtual Reality Technology ◽

Oculus Rift

Download Full-text

Graduate Curriculum for Biological Information Specialists: A Key to Integration of Scale in Biology

International Journal of Digital Curation ◽

10.2218/ijdc.v2i2.27 ◽

2008 ◽

Vol 2 (2) ◽

pp. 31-40 ◽

Cited By ~ 6

Author(s):

Carole L. Palmer ◽

Bryan P. Heidorn ◽

Dan Wright ◽

Melissa H. Cragin

Keyword(s):

Biological Sciences ◽

Multiple Scales ◽

Scientific Information ◽

Scientific Data ◽

High Rate ◽

Biological Information ◽

Biological Research ◽

Multi Scale ◽

Graduate Curriculum ◽

Professional Capacity

Scientific data problems do not stand in isolation. They are part of a larger set of challenges associated with the escalation of scientific information and changes in scholarly communication in the digital environment. Biologists in particular are generating enormous sets of data at a high rate, and new discoveries in the biological sciences will increasingly depend on the integration of data across multiple scales. This work will require new kinds of information expertise in key areas. To build this professional capacity we have developed two complementary educational programs: a Biological Information Specialist (BIS) masters degree and a concentration in Data Curation (DC). We believe that BISs will be central in the development of cyberinfrastructure and information services needed to facilitate interdisciplinary and multi-scale science. Here we present three sample cases from our current research projects to illustrate areas in which we expect information specialists to make important contributions to biological research practice.

Download Full-text