scholarly journals Content-aware frame interpolation (CAFI): Deep Learning-based temporal super-resolution for fast bioimaging

2021 ◽  
Author(s):  
Martin Priessner ◽  
David C.A. Gaboriau ◽  
Arlo Sheridan ◽  
Tchern Lenn ◽  
Jonathan R. Chubb ◽  
...  
Keyword(s):  
Deep Learning ◽  
Light Exposure ◽  
Super Resolution ◽  
Single Particle Tracking ◽  
Video Frame ◽  
Learning Networks ◽  
Frame Interpolation ◽  
Cellular Processes ◽  
Image Pairs ◽  
Content Aware

The development of high-resolution microscopes has made it possible to investigate cellular processes in 4D (3D over time). However, observing fast cellular dynamics remains challenging as a consequence of photobleaching and phototoxicity. These issues become increasingly problematic with the depth of the volume acquired and the speed of the biological events of interest. Here, we report the implementation of two content-aware frame interpolation (CAFI) deep learning networks, Zooming SlowMo (ZS) and Depth-Aware Video Frame Interpolation (DAIN), based on combinations of recurrent neural networks, that are highly suited for accurately predicting images in between image pairs, therefore improving the temporal resolution of image series as a post-acquisition analysis step. We show that CAFI predictions are capable of understanding the motion context of biological structures to perform better than standard interpolation methods. We benchmark CAFI's performance on six different datasets, obtained from three different microscopy modalities (point-scanning confocal, spinning-disc confocal and confocal brightfield microscopy). We demonstrate its capabilities for single-particle tracking methods applied to the study of lysosome trafficking. CAFI therefore allows for reduced light exposure and phototoxicity on the sample and extends the possibility of long-term live-cell imaging. Both DAIN and ZS as well as the training and testing data are made available for use by the wider community via the ZeroCostDL4Mic platform.

scholarly journals Accelerating Super-Resolution and Visual Task Analysis in Medical Images

2020 ◽  
Vol 10 (12) ◽  
pp. 4282
Author(s):  
Ghada Zamzmi ◽  
Sivaramakrishnan Rajaraman ◽  
Sameer Antani
Keyword(s):  
Deep Learning ◽  
High Resolution ◽  
Task Analysis ◽  
Multiple Scales ◽  
Medical Images ◽  
Computational Cost ◽  
Super Resolution ◽  
Visual Task ◽  
Learning Networks ◽  
High Resolution Images

Medical images are acquired at different resolutions based on clinical goals or available technology. In general, however, high-resolution images with fine structural details are preferred for visual task analysis. Recognizing this significance, several deep learning networks have been proposed to enhance medical images for reliable automated interpretation. These deep networks are often computationally complex and require a massive number of parameters, which restrict them to highly capable computing platforms with large memory banks. In this paper, we propose an efficient deep learning approach, called Hydra, which simultaneously reduces computational complexity and improves performance. The Hydra consists of a trunk and several computing heads. The trunk is a super-resolution model that learns the mapping from low-resolution to high-resolution images. It has a simple architecture that is trained using multiple scales at once to minimize a proposed learning-loss function. We also propose to append multiple task-specific heads to the trained Hydra trunk for simultaneous learning of multiple visual tasks in medical images. The Hydra is evaluated on publicly available chest X-ray image collections to perform image enhancement, lung segmentation, and abnormality classification. Our experimental results support our claims and demonstrate that the proposed approach can improve the performance of super-resolution and visual task analysis in medical images at a remarkably reduced computational cost.

scholarly journals Reducing System Load of Effective Video Using a Network Model

2021 ◽  
Vol 11 (20) ◽  
pp. 9665
Author(s):  
Soo-Young Cho ◽  
Dae-Yeol Kim ◽  
Su-Yeong Oh ◽  
Chae-Bong Sohn
Keyword(s):  
Video Transmission ◽  
Super Resolution ◽  
Field Of View ◽  
Video Frame ◽  
Generative Adversarial Network ◽  
Frame Interpolation ◽  
System Load ◽  
Face To Face ◽  
Adversarial Network ◽  
Streaming Services

Recently, as non-face-to-face work has become more common, the development of streaming services has become a significant issue. As these services are applied in increasingly diverse fields, various problems are caused by the overloading of systems when users try to transmit high-quality images. In this paper, SRGAN (Super Resolution Generative Adversarial Network) and DAIN (Depth-Aware Video Frame Interpolation) deep learning were used to reduce the overload that occurs during real-time video transmission. Images were divided into a FoV (Field of view) region and a non-FoV (Non-Field of view) region, and SRGAN was applied to the former, DAIN to the latter. Through this process, image quality was improved and system load was reduced.

scholarly journals FISR: Deep Joint Frame Interpolation and Super-Resolution with a Multi-Scale Temporal Loss

2020 ◽  
Vol 34 (07) ◽  
pp. 11278-11286 ◽  
Author(s):  
Soo Ye Kim ◽  
Jihyong Oh ◽  
Munchurl Kim
Keyword(s):  
Video Sequence ◽  
Super Resolution ◽  
Motion Artifacts ◽  
Video Frame ◽  
High Definition ◽  
Frame Interpolation ◽  
Display Devices ◽  
Multi Scale ◽  
Training Scheme ◽  
Spatio Temporal

Super-resolution (SR) has been widely used to convert low-resolution legacy videos to high-resolution (HR) ones, to suit the increasing resolution of displays (e.g. UHD TVs). However, it becomes easier for humans to notice motion artifacts (e.g. motion judder) in HR videos being rendered on larger-sized display devices. Thus, broadcasting standards support higher frame rates for UHD (Ultra High Definition) videos (4K@60 fps, 8K@120 fps), meaning that applying SR only is insufficient to produce genuine high quality videos. Hence, to up-convert legacy videos for realistic applications, not only SR but also video frame interpolation (VFI) is necessitated. In this paper, we first propose a joint VFI-SR framework for up-scaling the spatio-temporal resolution of videos from 2K 30 fps to 4K 60 fps. For this, we propose a novel training scheme with a multi-scale temporal loss that imposes temporal regularization on the input video sequence, which can be applied to any general video-related task. The proposed structure is analyzed in depth with extensive experiments.

scholarly journals Dose Super-Resolution in Prostate Volumetric Modulated Arc Therapy Using Cascaded Deep Learning Networks

2020 ◽  
Vol 10 ◽  
Author(s):  
Dong-Seok Shin ◽  
Kyeong-Hyeon Kim ◽  
Sang-Won Kang ◽  
Seong-Hee Kang ◽  
Jae-Sung Kim ◽  
...  
Keyword(s):  
Deep Learning ◽  
Volumetric Modulated Arc Therapy ◽  
Super Resolution ◽  
Learning Networks ◽  
Arc Therapy

scholarly journals Block Based Enhancement using Deep Learning for Conversion of Low Resolution AVS Video to High Resolution HEVC Video

2021 ◽  
Author(s):  
Rizwan Qureshi ◽  
Mehmood Nawaz
Keyword(s):  
Deep Learning ◽  
High Efficiency ◽  
Motion Vector ◽  
Super Resolution ◽  
Computational Time ◽  
Video Frame ◽  
Low Resolution ◽  
High Efficiency Video Coding ◽  
High Definition ◽  
Resolution Method

Conversion of one video bitstream to another video bitstream is a challenging task in the heterogeneous transcoder due to different video formats. In this paper, a region of interest (ROI) based super resolution technique is used to convert the lowresolution AVS (audio video standard) video to high definition HEVC (high efficiency video coding) video. Firstly, we classify a low-resolution video frame into small blocks by using visual characteristics, transform coefficients, and motion vector (MV) of a video. These blocks are further classified as blocks of most interest (BOMI), blocks of less interest (BOLI) and blocks of noninterest (BONI). The BONI blocks are considered as background blocks due to less interest in video and remains unchanged during SR process. Secondly, we apply deep learning based super resolution method on low resolution BOMI, and BOLI blocks to enhance the visual quality. The BOMI and BOLI blocks have high attention due to ROI that include some motion and contrast of the objects. The proposed method saves 20% to 30% computational time and obtained appreciable results as compared with full frame based super resolution method. We have tested our method on different official video sequences with resolution of 1K, 2K, and 4K. Our proposed method has an efficient visual performance in contrast to the full frame-based super resolution method.

scholarly journals Block Based Enhancement using Deep Learning for Conversion of Low Resolution AVS Video to High Resolution HEVC Video

2021 ◽  
Author(s):  
Rizwan Qureshi ◽  
Mehmood Nawaz
Keyword(s):  
Deep Learning ◽  
High Efficiency ◽  
Motion Vector ◽  
Super Resolution ◽  
Computational Time ◽  
Video Frame ◽  
Low Resolution ◽  
High Efficiency Video Coding ◽  
High Definition ◽  
Resolution Method

Conversion of one video bitstream to another video bitstream is a challenging task in the heterogeneous transcoder due to different video formats. In this paper, a region of interest (ROI) based super resolution technique is used to convert the lowresolution AVS (audio video standard) video to high definition HEVC (high efficiency video coding) video. Firstly, we classify a low-resolution video frame into small blocks by using visual characteristics, transform coefficients, and motion vector (MV) of a video. These blocks are further classified as blocks of most interest (BOMI), blocks of less interest (BOLI) and blocks of noninterest (BONI). The BONI blocks are considered as background blocks due to less interest in video and remains unchanged during SR process. Secondly, we apply deep learning based super resolution method on low resolution BOMI, and BOLI blocks to enhance the visual quality. The BOMI and BOLI blocks have high attention due to ROI that include some motion and contrast of the objects. The proposed method saves 20% to 30% computational time and obtained appreciable results as compared with full frame based super resolution method. We have tested our method on different official video sequences with resolution of 1K, 2K, and 4K. Our proposed method has an efficient visual performance in contrast to the full frame-based super resolution method.

scholarly journals Performance of deep learning restoration methods for the extraction of particle dynamics in noisy microscopy image sequences

2021 ◽  
pp. mbc.E20-11-0689
Author(s):  
Paul Kefer ◽  
Fadil Iqbal ◽  
Maelle Locatelli ◽  
Josh Lawrimore ◽  
Mengdi Zhang ◽  
...  
Keyword(s):  
Deep Learning ◽  
Image Restoration ◽  
Particle Tracking ◽  
Light Exposure ◽  
Image Sequences ◽  
Single Particle Tracking ◽  
Short Exposure ◽  
Tracking Accuracy ◽  
Excitation Light ◽  
Noisy Images

Particle tracking in living systems requires low light exposure and short exposure times to avoid phototoxicity and photobleaching and to fully capture particle motion with high-speed imaging. Low excitation light comes at the expense of tracking accuracy. Image restoration methods based on deep learning dramatically improve the signal-to-noise ratio in low-exposure datasets, qualitatively improving the images. However, it is not clear whether images generated by these methods yield accurate quantitative measurements such as diffusion parameters in (single) particle tracking experiments. Here, we evaluate the performance of two popular deep learning denoising software packages for particle tracking, using synthetic datasets and movies of diffusing chromatin as biological examples. With synthetic data, both supervised and unsupervised deep learning restored particle motions with high accuracy in two-dimensional datasets, whereas artifacts were introduced by the denoisers in 3D datasets. Experimentally, we found that, while both supervised and unsupervised approaches improved tracking results compared to the original noisy images, supervised learning generally outperformed the unsupervised approach. We find that nicer-looking image sequences are not synonymous with more precise tracking results and highlight that deep learning algorithms can produce deceiving artifacts with extremely noisy images. Finally, we address the challenge of selecting parameters to train convolutional neural networks by implementing a frugal Bayesian optimizer that rapidly explores multidimensional parameter spaces, identifying networks yielding optimal particle tracking accuracy. Our study provides quantitative outcome measures of image restoration using deep learning. We anticipate broad application of this approach to critically evaluate artificial intelligence solutions for quantitative microscopy.

scholarly journals Performance of deep learning restoration methods for the extraction of particle dynamics in noisy microscopy image sequences

2020 ◽  
Author(s):  
Paul Kefer ◽  
Fadil Iqbal ◽  
Maelle Locatelli ◽  
Josh Lawrimore ◽  
Mengdi Zhang ◽  
...  
Keyword(s):  
Deep Learning ◽  
Image Restoration ◽  
Particle Tracking ◽  
Cell Biology ◽  
Light Exposure ◽  
Image Sequences ◽  
Single Particle Tracking ◽  
Short Exposure ◽  
Tracking Accuracy ◽  
Quantitative Microscopy

ABSTRACTImage-based particle tracking is an essential tool to answer research questions in cell biology and beyond. A major challenge of particle tracking in living systems is that low light exposure is required to avoid phototoxicity and photobleaching. In addition, high-speed imaging used to fully capture particle motion dictates fast image acquisition rates. Short exposure times come at the expense of tracking accuracy. This is generally true for quantitative microscopy approaches and particularly relevant to single molecule tracking where the number of photons emitted from a single chromophore is limited. Image restoration methods based on deep learning dramatically improve the signal-to-noise ratio in low-exposure datasets. However, it is not clear whether images generated by these methods yield accurate quantitative measurements such as diffusion parameters in (single) particle tracking experiments. Here, we evaluate the performance of two popular deep learning denoising software packages for particle tracking, using synthetic datasets and movies of diffusing chromatin as biological examples. With synthetic data, both supervised and unsupervised deep learning restored particle motions with high accuracy in two-dimensional datasets, whereas artifacts were introduced by the denoisers in 3D datasets. Experimentally, we found that, while both supervised and unsupervised approaches improved the number of trackable particles and tracking accuracy, supervised learning generally outperformed the unsupervised approach, as expected. We also highlight that with extremely noisy image sequences, deep learning algorithms produce deceiving artifacts, which underscores the need to carefully evaluate the results. Finally, we address the challenge of selecting hyper-parameters to train convolutional neural networks by implementing a frugal Bayesian optimizer that rapidly explores multidimensional parameter spaces, identifying networks yielding optional particle tracking accuracy. Our study provides quantitative outcome measures of image restoration using deep learning. We anticipate broad application of the approaches presented here to critically evaluate artificial intelligence solutions for quantitative microscopy.

scholarly journals Analysis of Coronary Angiography Video Interpolation Methods to Reduce X-ray Exposure Frequency Based on Deep Learning

2021 ◽  
Author(s):  
Dong-xue Liang
Keyword(s):  
Deep Learning ◽  
Coronary Angiography ◽  
Coronary Arteries ◽  
Heart Surgery ◽  
Frame Rate ◽  
Video Frame ◽  
X Rays ◽  
Frame Interpolation ◽  
X Ray ◽  
Video Frame Rate

Cardiac coronary angiography is a major technique that assists physicians during interventional heart surgery. Under X-ray irradiation, the physician injects a contrast agent through a catheter and determines the coronary arteries’ state in real time. However, to obtain a more accurate state of the coronary arteries, physicians need to increase the frequency and intensity of X-ray exposure, which will inevitably increase the potential for harm to both the patient and the surgeon. In the work reported here, we use advanced deep learning algorithms to find a method of frame interpolation for coronary angiography videos that reduces the frequency of X-ray exposure by reducing the frame rate of the coronary angiography video, thereby reducing X-ray-induced damage to physicians. We established a new coronary angiography image group dataset containing 95,039 groups of images extracted from 31 videos. Each group includesthree consecutive images, which are used to train the video interpolation network model. We apply six popular frameinterpolation methods to this dataset to confirm that the video frame interpolation technology can reduce the video frame rate and reduce exposure of physicians to X-rays.

Sign in / Sign up

Export Citation Format

Share Document