scholarly journals Rectal Cancer Treatment Management: Deep-Learning Neural Network Based on Photoacoustic Microscopy Image Outperforms Histogram-Feature-Based Classification

2021 ◽  
Vol 11 ◽  
Author(s):  
Xiandong Leng ◽  
Eghbal Amidi ◽  
Sitai Kou ◽  
Hassam Cheema ◽  
Ebunoluwa Otegbeye ◽  
...  
Keyword(s):  
Neural Network ◽  
Rectal Cancer ◽  
Deep Learning ◽  
Linear Models ◽  
Ex Vivo ◽  
Residual Tumor ◽  
Photoacoustic Microscopy ◽  
Feature Based ◽  
Histogram Feature

We have developed a novel photoacoustic microscopy/ultrasound (PAM/US) endoscope to image post-treatment rectal cancer for surgical management of residual tumor after radiation and chemotherapy. Paired with a deep-learning convolutional neural network (CNN), the PAM images accurately differentiated pathological complete responders (pCR) from incomplete responders. However, the role of CNNs compared with traditional histogram-feature based classifiers needs further exploration. In this work, we compare the performance of the CNN models to generalized linear models (GLM) across 24 ex vivo specimens and 10 in vivo patient examinations. First order statistical features were extracted from histograms of PAM and US images to train, validate and test GLM models, while PAM and US images were directly used to train, validate, and test CNN models. The PAM-CNN model performed superiorly with an AUC of 0.96 (95% CI: 0.95-0.98) compared to the best PAM-GLM model using kurtosis with an AUC of 0.82 (95% CI: 0.82-0.83). We also found that both CNN and GLMs derived from photoacoustic data outperformed those utilizing ultrasound alone. We conclude that deep-learning neural networks paired with photoacoustic images is the optimal analysis framework for determining presence of residual cancer in the treated human rectum.

scholarly journals Biopsy-free in vivo virtual histology of skin using deep learning

2021 ◽  
Vol 10 (1) ◽  
Author(s):  
Jingxi Li ◽  
Jason Garfinkel ◽  
Xiaoran Zhang ◽  
Di Wu ◽  
Yijie Zhang ◽  
...  
Keyword(s):  
Neural Network ◽  
Deep Learning ◽  
Ex Vivo ◽  
Normal Skin ◽  
Ground Truth ◽  
Skin Neoplasms ◽  
Cellular Level ◽  
Label Free ◽  
Virtual Histology

AbstractAn invasive biopsy followed by histological staining is the benchmark for pathological diagnosis of skin tumors. The process is cumbersome and time-consuming, often leading to unnecessary biopsies and scars. Emerging noninvasive optical technologies such as reflectance confocal microscopy (RCM) can provide label-free, cellular-level resolution, in vivo images of skin without performing a biopsy. Although RCM is a useful diagnostic tool, it requires specialized training because the acquired images are grayscale, lack nuclear features, and are difficult to correlate with tissue pathology. Here, we present a deep learning-based framework that uses a convolutional neural network to rapidly transform in vivo RCM images of unstained skin into virtually-stained hematoxylin and eosin-like images with microscopic resolution, enabling visualization of the epidermis, dermal-epidermal junction, and superficial dermis layers. The network was trained under an adversarial learning scheme, which takes ex vivo RCM images of excised unstained/label-free tissue as inputs and uses the microscopic images of the same tissue labeled with acetic acid nuclear contrast staining as the ground truth. We show that this trained neural network can be used to rapidly perform virtual histology of in vivo, label-free RCM images of normal skin structure, basal cell carcinoma, and melanocytic nevi with pigmented melanocytes, demonstrating similar histological features to traditional histology from the same excised tissue. This application of deep learning-based virtual staining to noninvasive imaging technologies may permit more rapid diagnoses of malignant skin neoplasms and reduce invasive skin biopsies.

scholarly journals Real-time deep learning semantic segmentation during intra-operative surgery for 3D augmented reality assistance

2021 ◽  
Author(s):  
Leonardo Tanzi ◽  
Pietro Piazzolla ◽  
Francesco Porpiglia ◽  
Enrico Vezzetti
Keyword(s):  
Neural Network ◽  
Deep Learning ◽  
Augmented Reality ◽  
Ad Hoc ◽  
Geodesic Distance ◽  
Semantic Segmentation ◽  
Endoscopic Image ◽  
Operative Surgery ◽  
Endoscopic Videos

Abstract Purpose The current study aimed to propose a Deep Learning (DL) and Augmented Reality (AR) based solution for a in-vivo robot-assisted radical prostatectomy (RARP), to improve the precision of a published work from our group. We implemented a two-steps automatic system to align a 3D virtual ad-hoc model of a patient’s organ with its 2D endoscopic image, to assist surgeons during the procedure. Methods This approach was carried out using a Convolutional Neural Network (CNN) based structure for semantic segmentation and a subsequent elaboration of the obtained output, which produced the needed parameters for attaching the 3D model. We used a dataset obtained from 5 endoscopic videos (A, B, C, D, E), selected and tagged by our team’s specialists. We then evaluated the most performing couple of segmentation architecture and neural network and tested the overlay performances. Results U-Net stood out as the most effecting architectures for segmentation. ResNet and MobileNet obtained similar Intersection over Unit (IoU) results but MobileNet was able to elaborate almost twice operations per seconds. This segmentation technique outperformed the results from the former work, obtaining an average IoU for the catheter of 0.894 (σ = 0.076) compared to 0.339 (σ = 0.195). This modifications lead to an improvement also in the 3D overlay performances, in particular in the Euclidean Distance between the predicted and actual model’s anchor point, from 12.569 (σ= 4.456) to 4.160 (σ = 1.448) and in the Geodesic Distance between the predicted and actual model’s rotations, from 0.266 (σ = 0.131) to 0.169 (σ = 0.073). Conclusion This work is a further step through the adoption of DL and AR in the surgery domain. In future works, we will overcome the limits of this approach and finally improve every step of the surgical procedure.

scholarly journals Deep-learning-based motion-correction algorithm in optical resolution photoacoustic microscopy

2019 ◽  
Vol 2 (1) ◽  
Author(s):  
Xingxing Chen ◽  
Weizhi Qi ◽  
Lei Xi
Keyword(s):  
Deep Learning ◽  
Optical Resolution ◽  
Motion Artifacts ◽  
Photoacoustic Microscopy ◽  
Simulation Studies ◽  
Correction Algorithm ◽  
Brain Vessels ◽  
Capillary Networks ◽  
Motion Corrections

Abstract In this study, we propose a deep-learning-based method to correct motion artifacts in optical resolution photoacoustic microscopy (OR-PAM). The method is a convolutional neural network that establishes an end-to-end map from input raw data with motion artifacts to output corrected images. First, we performed simulation studies to evaluate the feasibility and effectiveness of the proposed method. Second, we employed this method to process images of rat brain vessels with multiple motion artifacts to evaluate its performance for in vivo applications. The results demonstrate that this method works well for both large blood vessels and capillary networks. In comparison with traditional methods, the proposed method in this study can be easily modified to satisfy different scenarios of motion corrections in OR-PAM by revising the training sets.

scholarly journals A structure-based deep learning framework for protein engineering

2019 ◽  
Author(s):  
Raghav Shroff ◽  
Austin W. Cole ◽  
Barrett R. Morrow ◽  
Daniel J. Diaz ◽  
Isaac Donnell ◽  
...  
Keyword(s):  
Neural Network ◽  
Amino Acids ◽  
Deep Learning ◽  
Protein Engineering ◽  
Convolutional Neural Network ◽  
State Of The Art ◽  
A Priori ◽  
Novel Proteins ◽  
Learning Framework

AbstractWhile deep learning methods exist to guide protein optimization, examples of novel proteins generated with these techniques require a priori mutational data. Here we report a 3D convolutional neural network that associates amino acids with neighboring chemical microenvironments at state-of-the-art accuracy. This algorithm enables identification of novel gain-of-function mutations, and subsequent experiments confirm substantive phenotypic improvements in stability-associated phenotypes in vivo across three diverse proteins.

Different molecular enumeration influences in deep learning: an example using aqueous solubility

2020 ◽  
Author(s):  
Jen-Hao Chen ◽  
Yufeng Jane Tseng
Keyword(s):  
Neural Network ◽  
Deep Learning ◽  
Single Molecule ◽  
Chemical Structure ◽  
Molecular Graph ◽  
Aqueous Solubility ◽  
Solubility Prediction ◽  
Input Line ◽  
Biological Phenomena ◽  
Feature Based

Abstract Aqueous solubility is the key property driving many chemical and biological phenomena and impacts experimental and computational attempts to assess those phenomena. Accurate prediction of solubility is essential and challenging, even with modern computational algorithms. Fingerprint-based, feature-based and molecular graph-based representations have all been used with different deep learning methods for aqueous solubility prediction. It has been clearly demonstrated that different molecular representations impact the model prediction and explainability. In this work, we reviewed different representations and also focused on using graph and line notations for modeling. In general, one canonical chemical structure is used to represent one molecule when computing its properties. We carefully examined the commonly used simplified molecular-input line-entry specification (SMILES) notation representing a single molecule and proposed to use the full enumerations in SMILES to achieve better accuracy. A convolutional neural network (CNN) was used. The full enumeration of SMILES can improve the presentation of a molecule and describe the molecule with all possible angles. This CNN model can be very robust when dealing with large datasets since no additional explicit chemistry knowledge is necessary to predict the solubility. Also, traditionally it is hard to use a neural network to explain the contribution of chemical substructures to a single property. We demonstrated the use of attention in the decoding network to detect the part of a molecule that is relevant to solubility, which can be used to explain the contribution from the CNN.

scholarly journals Direct pixel to pixel principal strain mapping from tagging MRI using end to end deep convolutional neural network (DeepStrain)

2021 ◽  
Vol 11 (1) ◽  
Author(s):  
Khaled Z. Abd-Elmoniem ◽  
Inas A. Yassine ◽  
Nader S. Metwalli ◽  
Ahmed Hamimi ◽  
Ronald Ouwerkerk ◽  
...  
Keyword(s):  
Neural Network ◽  
Monte Carlo ◽  
Deep Learning ◽  
Convolutional Neural Network ◽  
Ground Truth ◽  
Image Resolution ◽  
Principal Strain ◽  
Strain Mapping ◽  
End To End

AbstractRegional soft tissue mechanical strain offers crucial insights into tissue's mechanical function and vital indicators for different related disorders. Tagging magnetic resonance imaging (tMRI) has been the standard method for assessing the mechanical characteristics of organs such as the heart, the liver, and the brain. However, constructing accurate artifact-free pixelwise strain maps at the native resolution of the tagged images has for decades been a challenging unsolved task. In this work, we developed an end-to-end deep-learning framework for pixel-to-pixel mapping of the two-dimensional Eulerian principal strains $$\varvec{{\varepsilon }}_{\boldsymbol{p1}}$$ ε p 1 and $$\varvec{{\varepsilon }}_{\boldsymbol{p2}}$$ ε p 2 directly from 1-1 spatial modulation of magnetization (SPAMM) tMRI at native image resolution using convolutional neural network (CNN). Four different deep learning conditional generative adversarial network (cGAN) approaches were examined. Validations were performed using Monte Carlo computational model simulations, and in-vivo datasets, and compared to the harmonic phase (HARP) method, a conventional and validated method for tMRI analysis, with six different filter settings. Principal strain maps of Monte Carlo tMRI simulations with various anatomical, functional, and imaging parameters demonstrate artifact-free solid agreements with the corresponding ground-truth maps. Correlations with the ground-truth strain maps were R = 0.90 and 0.92 for the best-proposed cGAN approach compared to R = 0.12 and 0.73 for the best HARP method for $$\varvec{{\varepsilon }}_{\boldsymbol{p1}}$$ ε p 1 and $$\varvec{{\varepsilon }}_{\boldsymbol{p2}}$$ ε p 2 , respectively. The proposed cGAN approach's error was substantially lower than the error in the best HARP method at all strain ranges. In-vivo results are presented for both healthy subjects and patients with cardiac conditions (Pulmonary Hypertension). Strain maps, obtained directly from their corresponding tagged MR images, depict for the first time anatomical, functional, and temporal details at pixelwise native high resolution with unprecedented clarity. This work demonstrates the feasibility of using the deep learning cGAN for direct myocardial and liver Eulerian strain mapping from tMRI at native image resolution with minimal artifacts.

scholarly journals Investigating and Assessing the Dermoepidermal Junction with Multiphoton Microscopy and Deep Learning

2019 ◽  
Author(s):  
Mikko J. Huttunen ◽  
Radu Hristu ◽  
Adrian Dumitru ◽  
Mariana Costache ◽  
Stefan G. Stanciu
Keyword(s):  
Deep Learning ◽  
Human Error ◽  
Ex Vivo ◽  
Multiphoton Microscopy ◽  
Skin Cancer Screening ◽  
Dermoepidermal Junction ◽  
Optical Signals ◽  
Skin Cancers ◽  
Histopathological Image

AbstractHistopathological image analysis performed by a trained expert is currently regarded as the gold-standard in the case of many pathologies, including cancers. However, such approaches are laborious, time consuming and contain a risk for bias or human error. There is thus a clear need for faster, less intrusive and more accurate diagnostic solutions, requiring also minimal human intervention. Multiphoton Microscopy (MPM) can alleviate some of the drawbacks specific to traditional histopathology by exploiting various endogenous optical signals to provide virtual biopsies that reflect the architecture and composition of tissues, both in-vivo or ex-vivo. Here we show that MPM imaging of the dermoepidermal junction (DEJ) in unstained tissues provides useful cues for a histopathologist to identify the onset of non-melanoma skin cancers. Furthermore, we show that MPM images collected on the DEJ, besides being easy to interpret by a trained specialist, can be automatically classified into healthy and dysplastic classes with high precision using a Deep Learning method and existing pre-trained Convolutional Neural Networks. Our results suggest that Deep Learning enhanced MPM for in-vivo skin cancer screening could facilitate timely diagnosis and intervention, enabling thus more optimal therapeutic approaches.

Sign in / Sign up

Export Citation Format

Share Document