scholarly journals Generative adversarial networks for reconstructing natural images from brain activity

NeuroImage ◽  
2018 ◽  
Vol 181 ◽  
pp. 775-785 ◽  
Author(s):  
K. Seeliger ◽  
U. Güçlü ◽  
L. Ambrogioni ◽  
Y. Güçlütürk ◽  
M.A.J. van Gerven
Keyword(s):  
Brain Activity ◽  
Natural Images ◽  
Generative Adversarial Networks ◽  
Adversarial Networks

scholarly journals Smooth Deep Image Generator from Noises

2019 ◽  
Vol 33 ◽  
pp. 3731-3738 ◽  
Author(s):  
Tianyu Guo ◽  
Chang Xu ◽  
Boxin Shi ◽  
Chao Xu ◽  
Dacheng Tao
Keyword(s):  
Linear Interpolation ◽  
Natural Images ◽  
Generative Adversarial Networks ◽  
High Quality ◽  
Adversarial Networks ◽  
Deep Image ◽  
Image Generator ◽  
Input Perturbation ◽  
World Image ◽  
Strong Ability

Generative Adversarial Networks (GANs) have demonstrated a strong ability to fit complex distributions since they were presented, especially in the field of generating natural images. Linear interpolation in the noise space produces a continuously changing in the image space, which is an impressive property of GANs. However, there is no special consideration on this property in the objective function of GANs or its derived models. This paper analyzes the perturbation on the input of the generator and its influence on the generated images. A smooth generator is then developed by investigating the tolerable input perturbation. We further integrate this smooth generator with a gradient penalized discriminator, and design smooth GAN that generates stable and high-quality images. Experiments on real-world image datasets demonstrate the necessity of studying smooth generator and the effectiveness of the proposed algorithm.

scholarly journals Human Sensitivity to Perturbations Constrained by a Model of the Natural Image Manifold

2018 ◽  
Author(s):  
Ingo Fruend ◽  
Elee Stalker
Keyword(s):  
Image Segmentation ◽  
Statistical Properties ◽  
Natural Images ◽  
Generative Adversarial Networks ◽  
Natural Image ◽  
Quantitative Characterization ◽  
Adversarial Networks ◽  
Image Manifold ◽  
Visual Structure

Humans are remarkably well tuned to the statistical properties of natural images. However, quantitative characterization of processing within the domain of natural images has been difficult because most parametric manipulations of a natural image make that image appear less natural. We used generative adversarial networks (GANs) to constrain parametric manipulations to remain within an approximation of the manifold of natural images. In the first experiment, 7 observers decided which one of two synthetic perturbed images matched a synthetic unperturbed comparison image. Observers were significantly more sensitive to perturbations that were constrained to an approximate manifold of natural images than they were to perturbations applied directly in pixel space. Trial by trial errors were consistent with the idea that these perturbations disrupt configural aspects of visual structure used in image segmentation. In a second experiment, 5 observers discriminated paths along the image manifold as recovered by the GAN. Observers were remarkably good at this task, confirming that observers were tuned to fairly detailed properties of an approximate manifold of natural images. We conclude that human tuning to natural images is more general than detecting deviations from natural appearance, and that humans have, to some extent, access to detailed interrelations between natural images.

scholarly journals Generative adversarial networks for reconstructing natural images from brain activity

2017 ◽  
Author(s):  
K. Seeliger ◽  
U. Güçlü ◽  
L. Ambrogioni ◽  
Y. Güçlütürk ◽  
M. A. J. van Gerven
Keyword(s):  
Brain Activity ◽  
Pairwise Comparison ◽  
Natural Images ◽  
Generative Adversarial Networks ◽  
Natural Image ◽  
Semantic Features ◽  
Generative Adversarial Network ◽  
Adversarial Network ◽  
Generative Modeling ◽  
Reconstruction Performance

AbstractWe explore a method for reconstructing visual stimuli from brain activity. Using large databases of natural images we trained a deep convolutional generative adversarial network capable of generating gray scale photos, similar to stimUli presented during two functional magnetic resonance imaging experiments. Using a linear model we learned to predict the generative model’s latent space from measured brain activity. The objective was to create an image similar to the presented stimulus image through the previously trained generator. Using this approach we were able to reconstruct structural and some semantic features of a proportion of the natural images sets. A behavioral test showed that subjects were capable of identifying a reconstruction of the original stimuhis in 67.2% and 66.4% of the cases in a pairwise comparison for the two natural image datasets respectively. our approach does not require end-to-end training of a large generative model on limited neuroimaging data. Rapid advances in generative modeling promise further improvements in reconstruction performance.

scholarly journals Generative Adversarial Networks Conditioned on Brain Activity Reconstruct Seen Images

2018 ◽  
Author(s):  
Ghislain St-Yves ◽  
Thomas Naselaris
Keyword(s):  
Brain Activity ◽  
Generative Model ◽  
Fmri Data ◽  
Generative Adversarial Networks ◽  
High Dimensional ◽  
Inference Problem ◽  
Adversarial Networks ◽  
Neural Representations ◽  
Activity Measurements ◽  
Image Details

AbstractWe consider the inference problem of reconstructing a visual stimulus from brain activity measurements (e.g. fMRI) that encode this stimulus. Recovering a complete image is complicated by the fact that neural representations are noisy, high-dimensional, and contain incomplete information about image details. Thus, reconstructions of complex images from brain activity require a strong prior. Here we propose to train generative adversarial networks (GANs) to learn a generative model of images that is conditioned on measurements of brain activity. We consider two challenges of this approach: First, given that GANs require far more data to train than is typically collected in an fMRI experiment, how do we obtain enough samples to train a GAN that is conditioned on brain activity? Secondly, how do we ensure that our generated samples are robust against noise present in fMRI data? Our strategy to surmount both of these problems centers around the creation of surrogate brain activity samples that are generated by an encoding model. We find that the generative model thus trained generalizes to real fRMI data measured during perception of images and is able to reconstruct the basic outline of the stimuli.

Sign in / Sign up

Export Citation Format

Share Document