Harnessing Optoelectronic Noises in a Hybrid Photonic Generative Adversarial Network (GAN)

Abstract Integrated programmable optoelectronics is emerging as a promising platform of neural network accelerator, which affords efficient in-memory computing and high bandwidth interconnectivity. The analog nature of optical computing and the inherent optoelectronic noises, however, make the systems error-prone in practical implementations such as classification by discriminative neural networks. It is thus imperative to devise strategies to mitigate and, if possible, harness optical and electrical noises in optoelectronic computing systems. Here, we demonstrate a prototypical hybrid photonic generative adversarial network (GAN) that generates handwritten numbers using an optoelectronic core consisting of an array of programable phase-change optical memory cells. We harness optoelectronic noises in the hybrid photonic GAN by realizing an optoelectronic random number generator derived from the amplified spontaneous emission noise, applying noise-aware training by injecting additional noise to the network, and implementing the trained network with resilience to hardware non-idealities. Surprisingly, the hybrid photonic GAN with hardware noises and inaccuracies can generate images of even higher quality than the noiseless software baseline. Our results suggest the resilience and potential of more complex hybrid generative networks based on large-scale, non-ideal optoelectronic hardware. The demonstrated GAN architecture and the proposed noise-aware training approach are generic and thus applicable to various types of optoelectronic neuromorphic computing hardware.

Download Full-text

GANai: Standardizing CT Images using Generative Adversarial Network with Alternative Improvement

10.1101/460188 ◽

2018 ◽

Author(s):

Gongbo Liang ◽

Sajjad Fouladvand ◽

Jie Zhang ◽

Michael A. Brooks ◽

Nathan Jacobs ◽

...

Keyword(s):

Large Scale ◽

Model Performance ◽

Image Synthesis ◽

Ct Images ◽

Image Features ◽

Training Data ◽

Ct Image ◽

Generative Adversarial Network ◽

Training Strategy ◽

Adversarial Network

AbstractComputed tomography (CT) is a widely-used diag-reproducibility regarding radiomic features, such as intensity, nostic image modality routinely used for assessing anatomical tissue characteristics. However, non-standardized imaging pro-tocols are commonplace, which poses a fundamental challenge in large-scale cross-center CT image analysis. One approach to address the problem is to standardize CT images using generative adversarial network models (GAN). GAN learns the data distribution of training images and generate synthesized images under the same distribution. However, existing GAN models are not directly applicable to this task mainly due to the lack of constraints on the mode of data to generate. Furthermore, they treat every image equally, but in real applications, some images are more difficult to standardize than the others. All these may lead to the lack-of-detail problem in CT image synthesis. We present a new GAN model called GANai to mitigate the differences in radiomic features across CT images captured using non-standard imaging protocols. Given source images, GANai composes new images by specifying a high-level goal that the image features of the synthesized images should be similar to those of the standard images. GANai introduces an alternative improvement training strategy to alternatively and steadily improve model performance. The new training strategy enables a series of technical improvements, including phase-specific loss functions, phase-specific training data, and the adoption of ensemble learning, leading to better model performance. The experimental results show that GANai is significantly better than the existing state-of-the-art image synthesis algorithms on CT image standardization. Also, it significantly improves the efficiency and stability of GAN model training.

Download Full-text

Multi-omics Data Integration by Generative Adversarial Network

10.1101/2021.03.13.435251 ◽

2021 ◽

Author(s):

Khandakar Tanvir Ahmed ◽

Jiao Sun ◽

Jeongsik Yong ◽

Wei Zhang

Keyword(s):

Large Scale ◽

Synthetic Data ◽

Interaction Network ◽

Vital Role ◽

The Cancer Genome Atlas ◽

Survival Prediction ◽

Omics Data ◽

Generative Adversarial Network ◽

Adversarial Network ◽

Cancer Outcome

Accurate disease phenotype prediction plays an important role in the treatment of heterogeneous diseases like cancer in the era of precision medicine. With the advent of high throughput technologies, more comprehensive multi-omics data is now available that can effectively link the genotype to phenotype. However, the interactive relation of multi-omics datasets makes it particularly challenging to incorporate different biological layers to discover the coherent biological signatures and predict phenotypic outcomes. In this study, we introduce omicsGAN, a generative adversarial network (GAN) model to integrate two omics data and their interaction network. The model captures information from the interaction network as well as the two omics datasets and fuse them to generate synthetic data with better predictive signals. Large-scale experiments on The Cancer Genome Atlas (TCGA) breast cancer and ovarian cancer datasets validate that (1) the model can effectively integrate two omics data (i.e., mRNA and microRNA expression data) and their interaction network (i.e., microRNA-mRNA interaction network). The synthetic omics data generated by the proposed model has a better performance on cancer outcome classification and patients survival prediction compared to original omics datasets. (2) The integrity of the interaction network plays a vital role in the generation of synthetic data with higher predictive quality. Using a random interaction network does not allow the framework to learn meaningful information from the omics datasets; therefore, results in synthetic data with weaker predictive signals.

Download Full-text

IMAGE TO POINT CLOUD TRANSLATION USING CONDITIONAL GENERATIVE ADVERSARIAL NETWORK FOR AIRBORNE LIDAR DATA

ISPRS Annals of Photogrammetry Remote Sensing and Spatial Information Sciences ◽

10.5194/isprs-annals-v-2-2021-169-2021 ◽

2021 ◽

Vol V-2-2021 ◽

pp. 169-174

Author(s):

T. Shinohara ◽

H. Xiu ◽

M. Matsuoka

Keyword(s):

Point Cloud ◽

Large Scale ◽

Point Clouds ◽

Airborne Lidar ◽

Real Point ◽

3D Point Cloud ◽

Generative Adversarial Network ◽

Adversarial Network ◽

3D Point Clouds ◽

Latent Features

Abstract. This study introduces a novel image to a 3D point-cloud translation method with a conditional generative adversarial network that creates a large-scale 3D point cloud. This can generate supervised point clouds observed via airborne LiDAR from aerial images. The network is composed of an encoder to produce latent features of input images, generator to translate latent features to fake point clouds, and discriminator to classify false or real point clouds. The encoder is a pre-trained ResNet; to overcome the difficulty of generating 3D point clouds in an outdoor scene, we use a FoldingNet with features from ResNet. After a fixed number of iterations, our generator can produce fake point clouds that correspond to the input image. Experimental results show that our network can learn and generate certain point clouds using the data from the 2018 IEEE GRSS Data Fusion Contest.

Download Full-text

Cross-Modality Person Re-Identification with Generative Adversarial Training

Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence ◽

10.24963/ijcai.2018/94 ◽

2018 ◽

Cited By ~ 31

Author(s):

Pingyang Dai ◽

Rongrong Ji ◽

Haibin Wang ◽

Qiong Wu ◽

Yuyu Huang

Keyword(s):

Large Scale ◽

Characteristic Curve ◽

Metric Learning ◽

Feature Representation ◽

Superior Performance ◽

Generative Adversarial Network ◽

Adversarial Network ◽

Discriminative Feature ◽

Adversarial Training ◽

Rgb Images

Person re-identification (Re-ID) is an important task in video surveillance which automatically searches and identifies people across different cameras. Despite the extensive Re-ID progress in RGB cameras, few works have studied the Re-ID between infrared and RGB images, which is essentially a cross-modality problem and widely encountered in real-world scenarios. The key challenge lies in two folds, i.e., the lack of discriminative information to re-identify the same person between RGB and infrared modalities, and the difficulty to learn a robust metric towards such a large-scale cross-modality retrieval. In this paper, we tackle the above two challenges by proposing a novel cross-modality generative adversarial network (termed cmGAN). To handle the issue of insufficient discriminative information, we leverage the cutting-edge generative adversarial training to design our own discriminator to learn discriminative feature representation from different modalities. To handle the issue of large-scale cross-modality metric learning, we integrates both identification loss and cross-modality triplet loss, which minimize inter-class ambiguity while maximizing cross-modality similarity among instances. The entire cmGAN can be trained in an end-to-end manner by using standard deep neural network framework. We have quantized the performance of our work in the newly-released SYSU RGB-IR Re-ID benchmark, and have reported superior performance, i.e., Cumulative Match Characteristic curve (CMC) and Mean Average Precision (MAP), over the state-of-the-art works [Wu et al., 2017], respectively.

Download Full-text

GAN-GL: Generative Adversarial Networks for Glacial Lake Mapping

Remote Sensing ◽

10.3390/rs13224728 ◽

2021 ◽

Vol 13 (22) ◽

pp. 4728

Author(s):

Hang Zhao ◽

Meimei Zhang ◽

Fang Chen

Keyword(s):

Large Scale ◽

Glacial Lake ◽

Background Information ◽

Generative Adversarial Networks ◽

High Mountain ◽

Landsat 8 ◽

Glacial Lakes ◽

Ancillary Data ◽

Generative Adversarial Network ◽

Adversarial Network

Remote sensing is a powerful tool that provides flexibility and scalability for monitoring and investigating glacial lakes in High Mountain Asia (HMA). However, existing methods for mapping glacial lakes are designed based on a combination of several spectral features and ancillary data (such as the digital elevation model, DEM) to highlight the lake extent and suppress background information. These methods, however, suffer from either the inevitable requirement of post-processing work or the high costs of additional data acquisition. Signifying a key advancement in the deep learning models, a generative adversarial network (GAN) can capture multi-level features and learn the mapping rules in source and target domains using a minimax game between a generator and discriminator. This provides a new and feasible way to conduct large-scale glacial lake mapping. In this work, a complete glacial lake dataset was first created, containing approximately 4600 patches of Landsat-8 OLI images edited in three ways—random cropping, density cropping, and uniform cropping. Then, a GAN model for glacial lake mapping (GAN-GL) was constructed. The GAN-GL consists of two parts—a generator that incorporates a water attention module and an image segmentation module to produce the glacial lake masks, and a discriminator which employs the ResNet-152 backbone to ascertain whether a given pixel belonged to a glacial lake. The model was evaluated using the created glacial lake dataset, delivering a good performance, with an F1 score of 92.17% and IoU of 86.34%. Moreover, compared to the mapping results derived from the global–local iterative segmentation algorithm and random forest for the entire Eastern Himalayas, our proposed model was superior regarding the segmentation of glacial lakes under complex and diverse environmental conditions, in terms of accuracy (precision = 93.19%) and segmentation efficiency. Our model was also very good at detecting small glacial lakes without assistance from ancillary data or human intervention.

Download Full-text

Multifeature Named Entity Recognition in Information Security Based on Adversarial Learning

Security and Communication Networks ◽

10.1155/2019/6417407 ◽

2019 ◽

Vol 2019 ◽

pp. 1-9

Author(s):

Han Zhang ◽

Yuanbo Guo ◽

Tao Li

Keyword(s):

Information Security ◽

Large Scale ◽

Named Entity Recognition ◽

Entity Recognition ◽

Generative Adversarial Network ◽

Adversarial Learning ◽

New Approach ◽

Named Entity ◽

Adversarial Network ◽

Security Research

In order to obtain high quality and large-scale labelled data for information security research, we propose a new approach that combines a generative adversarial network with the BiLSTM-Attention-CRF model to obtain labelled data from crowd annotations. We use the generative adversarial network to find common features in crowd annotations and then consider them in conjunction with the domain dictionary feature and sentence dependency feature as additional features to be introduced into the BiLSTM-Attention-CRF model, which is then used to carry out named entity recognition in crowdsourcing. Finally, we create a dataset to evaluate our models using information security data. The experimental results show that our model has better performance than the other baseline models.

Download Full-text

Multi-Temporal Sentinel-1 and -2 Data Fusion for Optical Image Simulation

ISPRS International Journal of Geo-Information ◽

10.3390/ijgi7100389 ◽

2018 ◽

Vol 7 (10) ◽

pp. 389 ◽

Cited By ~ 19

Author(s):

Wei He ◽

Naoto Yokoya

Keyword(s):

Deep Learning ◽

Large Scale ◽

Super Resolution ◽

Optical Image ◽

Optical Data ◽

Image Simulation ◽

Generative Adversarial Network ◽

Adversarial Network ◽

Multi Temporal ◽

Sar Data

In this paper, we present the optical image simulation from synthetic aperture radar (SAR) data using deep learning based methods. Two models, i.e., optical image simulation directly from the SAR data and from multi-temporal SAR-optical data, are proposed to testify the possibilities. The deep learning based methods that we chose to achieve the models are a convolutional neural network (CNN) with a residual architecture and a conditional generative adversarial network (cGAN). We validate our models using the Sentinel-1 and -2 datasets. The experiments demonstrate that the model with multi-temporal SAR-optical data can successfully simulate the optical image; meanwhile, the state-of-the-art model with simple SAR data as input failed. The optical image simulation results indicate the possibility of SAR-optical information blending for the subsequent applications such as large-scale cloud removal, and optical data temporal super-resolution. We also investigate the sensitivity of the proposed models against the training samples, and reveal possible future directions.

Download Full-text

Extreme Low-Resolution Activity Recognition Using a Super-Resolution-Oriented Generative Adversarial Network

Micromachines ◽

10.3390/mi12060670 ◽

2021 ◽

Vol 12 (6) ◽

pp. 670

Author(s):

Mingzheng Hou ◽

Song Liu ◽

Jiliu Zhou ◽

Yi Zhang ◽

Ziliang Feng

Keyword(s):

Activity Recognition ◽

Large Scale ◽

Super Resolution ◽

General Activity ◽

Low Resolution ◽

Generative Adversarial Network ◽

Video Clips ◽

Adversarial Network ◽

Efficient Recognition ◽

Low Resolution Images

Activity recognition is a fundamental and crucial task in computer vision. Impressive results have been achieved for activity recognition in high-resolution videos, but for extreme low-resolution videos, which capture the action information at a distance and are vital for preserving privacy, the performance of activity recognition algorithms is far from satisfactory. The reason is that extreme low-resolution (e.g., 12 × 16 pixels) images lack adequate scene and appearance information, which is needed for efficient recognition. To address this problem, we propose a super-resolution-driven generative adversarial network for activity recognition. To fully take advantage of the latent information in low-resolution images, a powerful network module is employed to super-resolve the extremely low-resolution images with a large scale factor. Then, a general activity recognition network is applied to analyze the super-resolved video clips. Extensive experiments on two public benchmarks were conducted to evaluate the effectiveness of our proposed method. The results demonstrate that our method outperforms several state-of-the-art low-resolution activity recognition approaches.

Download Full-text

USING GENERATIVE ADVERSARIAL NETWORKS FOR EXTRACTION OF INSAR SIGNALS FROM LARGE-SCALE SENTINEL-1 INTERFEROGRAMS BY IMPROVING TROPOSPHERIC NOISE CORRECTION

ISPRS Annals of Photogrammetry Remote Sensing and Spatial Information Sciences ◽

10.5194/isprs-annals-v-3-2021-57-2021 ◽

2021 ◽

Vol V-3-2021 ◽

pp. 57-64

Author(s):

B. Ghosh ◽

M. Haghshenas Haghighi ◽

M. Motagh ◽

S. Maghsudi

Keyword(s):

Large Scale ◽

Ground Deformation ◽

Noise Model ◽

Generative Adversarial Networks ◽

Tropospheric Delay ◽

Generative Adversarial Network ◽

Water Vapour Content ◽

Adversarial Network ◽

Adversarial Networks ◽

Noise Correction

Abstract. Spatiotemporal variations of pressure, temperature, water vapour content in the atmosphere lead to significant delays in interferometric synthetic aperture radar (InSAR) measurements of deformations in the ground. One of the key challenges in increasing the accuracy of ground deformation measurements using InSAR is to produce robust estimates of the tropospheric delay. Tropospheric models like ERA-Interim can be used to estimate the total tropospheric delay in interferograms in remote areas. The problem with using ERA-Interim model for interferogram correction is that after the tropospheric correction, there are still some residuals left in the interferograms, which can be mainly attributed to turbulent troposphere. In this study, we propose a Generative Adversarial Network (GAN) based approach to mitigate the phase delay caused by troposphere. In this method, we implement a noise to noise model, where the network is trained only with the interferograms corrupted by tropospheric noise. We applied the technique over 116 large scale 800 km long interfergrams formed from Sentinel-1 acquisitions covering a period from 25th October, 2014 to 2nd November, 2017 from descending track numbered 108 over Iran. Our approach reduces the root mean square of the phase values of the interferogram 64% compared to those of the original interferogram and by 55% in comparison to the corresponding ERA-Interim corrected version.

Download Full-text

AWGAN: A Powerful Batch Correction Model for scRNA-seq Data

10.1101/2021.11.08.467781 ◽

2021 ◽

Author(s):

Tianyu Liu ◽

Yuge Wang ◽

Hong-yu Zhao

Keyword(s):

Deep Learning ◽

Large Scale ◽

Batch Effects ◽

Generative Adversarial Network ◽

Correction Model ◽

Batch Correction ◽

Learning Framework ◽

Adversarial Network ◽

Genomics Research ◽

Downstream Analysis

With the advancement of technology, we can generate and access large-scale, high dimensional and diverse genomics data, especially through single-cell RNA sequencing (scRNA-seq). However, integrative downstream analysis from multiple scRNA-seq datasets remains challenging due to batch effects. In this paper, we focus on scRNA-seq data integration and propose a new deep learning framework based on Wasserstein Generative Adversarial Network (WGAN) combined with an attention mechanism to reduce the differences among batches. We also discuss the limitations of the existing methods and demonstrate the advantages of our new model from both theoretical and practical aspects, advocating the use of deep learning in genomics research.

Download Full-text