scholarly journals Hyperrealistic neural decoding for reconstructing faces from fMRI activations via the GAN latent space

2022 ◽  
Vol 12 (1) ◽  
Author(s):  
Thirza Dado ◽  
Yağmur Güçlütürk ◽  
Luca Ambrogioni ◽  
Gabriëlle Ras ◽  
Sander Bosch ◽  
...  
Keyword(s):  
A Priori ◽  
Feature Space ◽  
Generative Adversarial Networks ◽  
Neural Decoding ◽  
Sensory Stimuli ◽  
Feature Representations ◽  
Adversarial Networks ◽  
Face Images ◽  
Brain Responses ◽  
Latent Space

AbstractNeural decoding can be conceptualized as the problem of mapping brain responses back to sensory stimuli via a feature space. We introduce (i) a novel experimental paradigm that uses well-controlled yet highly naturalistic stimuli with a priori known feature representations and (ii) an implementation thereof for HYPerrealistic reconstruction of PERception (HYPER) of faces from brain recordings. To this end, we embrace the use of generative adversarial networks (GANs) at the earliest step of our neural decoding pipeline by acquiring fMRI data as participants perceive face images synthesized by the generator network of a GAN. We show that the latent vectors used for generation effectively capture the same defining stimulus properties as the fMRI measurements. As such, these latents (conditioned on the GAN) are used as the in-between feature representations underlying the perceived images that can be predicted in neural decoding for (re-)generation of the originally perceived stimuli, leading to the most accurate reconstructions of perception to date.

scholarly journals A deep learning approach to capture the essence of Candida albicans morphologies

2021 ◽  
Author(s):  
Van Bettauer ◽  
Anna CBP Costa ◽  
Raha Parvizi Omran ◽  
Samira Massahi ◽  
Eftyhios Kirbizakis ◽  
...  
Keyword(s):  
Deep Learning ◽  
Learning Strategy ◽  
Developmental Trajectories ◽  
Generative Adversarial Networks ◽  
Learning Approach ◽  
Adversarial Networks ◽  
Latent Space ◽  
Contrast Microscopy ◽  
Opportunistic Human Pathogen ◽  
Community Meeting

We present deep learning-based approaches for exploring the complex array of morphologies exhibited by the opportunistic human pathogen C. albicans. Our system entitled Candescence automatically detects C. albicans cells from Differential Image Contrast microscopy, and labels each detected cell with one of nine vegetative, mating-competent or filamentous morphologies. The software is based upon a fully convolutional one-stage object detector and exploits a novel cumulative curriculum-based learning strategy that stratifies our images by difficulty from simple vegetative forms to more complex filamentous architectures. Candescence achieves very good performance on this difficult learning set which has substantial intermixing between the predicted classes. To capture the essence of each C. albicans morphology, we develop models using generative adversarial networks and identify subcomponents of the latent space which control technical variables, developmental trajectories or morphological switches. We envision Candescence as a community meeting point for quantitative explorations of C. albicans morphology.

scholarly journals Parallel Image Completion with Edge and Color Map

2019 ◽  
Vol 9 (18) ◽  
pp. 3856 ◽  
Author(s):  
Dan Zhao ◽  
Baolong Guo ◽  
Yunyi Yan
Keyword(s):  
Superior Performance ◽  
Generative Adversarial Networks ◽  
Image Completion ◽  
Significant Progress ◽  
Color Information ◽  
Parallel Edge ◽  
Adversarial Networks ◽  
Parallel Image ◽  
Latent Space ◽  
Image Edge

Over the last few years, image completion has made significant progress due to the generative adversarial networks (GANs) that are able to synthesize photorealistic contents. However, one of the main obstacles faced by many existing methods is that they often create blurry textures or distorted structures that are inconsistent with surrounding regions. The main reason is the ineffectiveness of disentangling style latent space implicitly from images. To address this problem, we develop a novel image completion framework called PIC-EC: parallel image completion networks with edge and color maps, which explicitly provides image edge and color information as the prior knowledge for image completion. The PIC-EC framework consists of the parallel edge and color generators followed by an image completion network. Specifically, the parallel paths generate edge and color maps for the missing region at the same time, and then the image completion network fills the missing region with fine details using the generated edge and color information as the priors. The proposed method was evaluated over CelebA-HQ and Paris StreetView datasets. Experimental results demonstrate that PIC-EC achieves superior performance on challenging cases with complex compositions and outperforms existing methods on evaluations of realism and accuracy, both quantitatively and qualitatively.

scholarly journals Age Progression and Regression with Spatial Attention Modules

2020 ◽  
Vol 34 (07) ◽  
pp. 11378-11385
Author(s):  
Qi Li ◽  
Yunfan Liu ◽  
Zhenan Sun
Keyword(s):  
Spatial Attention ◽  
Generative Adversarial Networks ◽  
Age Progression ◽  
Training Images ◽  
Adversarial Networks ◽  
Face Images ◽  
Multiple Datasets ◽  
Image Translation ◽  
In The Wild ◽  
Face Aging

Age progression and regression refers to aesthetically rendering a given face image to present effects of face aging and rejuvenation, respectively. Although numerous studies have been conducted in this topic, there are two major problems: 1) multiple models are usually trained to simulate different age mappings, and 2) the photo-realism of generated face images is heavily influenced by the variation of training images in terms of pose, illumination, and background. To address these issues, in this paper, we propose a framework based on conditional Generative Adversarial Networks (cGANs) to achieve age progression and regression simultaneously. Particularly, since face aging and rejuvenation are largely different in terms of image translation patterns, we model these two processes using two separate generators, each dedicated to one age changing process. In addition, we exploit spatial attention mechanisms to limit image modifications to regions closely related to age changes, so that images with high visual fidelity could be synthesized for in-the-wild cases. Experiments on multiple datasets demonstrate the ability of our model in synthesizing lifelike face images at desired ages with personalized features well preserved, and keeping age-irrelevant regions unchanged.

Synthesizing Designs With Interpart Dependencies Using Hierarchical Generative Adversarial Networks

2019 ◽  
Vol 141 (11) ◽  
Author(s):  
Wei Chen ◽  
Mark Fuge
Keyword(s):  
Constraint Satisfaction ◽  
Real World ◽  
Generative Models ◽  
Dependency Graph ◽  
Generative Adversarial Networks ◽  
Engineering Systems ◽  
Adversarial Networks ◽  
Latent Space ◽  
Branching Factor ◽  
Low Dimensional

Abstract Real-world designs usually consist of parts with interpart dependencies, i.e., the geometry of one part is dependent on one or multiple other parts. We can represent such dependency in a part dependency graph. This paper presents a method for synthesizing these types of hierarchical designs using generative models learned from examples. It decomposes the problem of synthesizing the whole design into synthesizing each part separately but keeping the interpart dependencies satisfied. Specifically, this method constructs multiple generative models, the interaction of which is based on the part dependency graph. We then use the trained generative models to synthesize or explore each part design separately via a low-dimensional latent representation, conditioned on the corresponding parent part(s). We verify our model on multiple design examples with different interpart dependencies. We evaluate our model by analyzing the constraint satisfaction performance, the synthesis quality, the latent space quality, and the effects of part dependency depth and branching factor. This paper’s techniques for capturing dependencies among parts lay the foundation for learned generative models to extend to more realistic engineering systems where such relationships are widespread.

scholarly journals Learning Representations of Natural Language Texts with Generative Adversarial Networks at Document, Sentence, and Aspect Level

Algorithms ◽  
2018 ◽  
Vol 11 (10) ◽  
pp. 164 ◽  
Author(s):  
Aggeliki Vlachostergiou ◽  
George Caridakis ◽  
Phivos Mylonas ◽  
Andreas Stafylopatis
Keyword(s):  
Natural Language ◽  
Generative Models ◽  
Generative Adversarial Networks ◽  
Real World Data ◽  
Generative Adversarial Network ◽  
Feature Representations ◽  
Adversarial Network ◽  
Adversarial Networks ◽  
Learning Tasks ◽  
Potential Applications

The ability to learn robust, resizable feature representations from unlabeled data has potential applications in a wide variety of machine learning tasks. One way to create such representations is to train deep generative models that can learn to capture the complex distribution of real-world data. Generative adversarial network (GAN) approaches have shown impressive results in producing generative models of images, but relatively little work has been done on evaluating the performance of these methods for the learning representation of natural language, both in supervised and unsupervised settings at the document, sentence, and aspect level. Extensive research validation experiments were performed by leveraging the 20 Newsgroups corpus, the Movie Review (MR) Dataset, and the Finegrained Sentiment Dataset (FSD). Our experimental analysis suggests that GANs can successfully learn representations of natural language texts at all three aforementioned levels.

scholarly journals A Deep Generative Model for Code Switched Text

2019 ◽  
Author(s):  
Bidisha Samanta ◽  
Sharmila Reddy ◽  
Hussain Jagirdar ◽  
Niloy Ganguly ◽  
Soumen Chakrabarti
Keyword(s):  
State Of The Art ◽  
Generative Models ◽  
Code Switching ◽  
Language Models ◽  
Generative Adversarial Networks ◽  
Data Intensive ◽  
Adversarial Networks ◽  
Latent Space ◽  
Variational Autoencoder ◽  
Multilingual Societies

Code-switching, the interleaving of two or more languages within a sentence or discourse is pervasive in multilingual societies. Accurate language models for code-switched text are critical for NLP tasks. State-of-the-art data-intensive neural language models are difficult to train well from scarce language-labeled code-switched text. A potential solution is to use deep generative models to synthesize large volumes of realistic code-switched text. Although generative adversarial networks and variational autoencoders can synthesize plausible monolingual text from continuous latent space, they cannot adequately address code-switched text, owing to their informal style and complex interplay between the constituent languages. We introduce VACS, a novel variational autoencoder architecture specifically tailored to code-switching phenomena. VACS encodes to and decodes from a two-level hierarchical representation, which models syntactic contextual signals in the lower level, and language switching signals in the upper layer. Sampling representations from the prior and decoding them produced well-formed, diverse code-switched sentences. Extensive experiments show that using synthetic code-switched text with natural monolingual data results in significant (33.06\%) drop in perplexity.

Sign in / Sign up

Export Citation Format

Share Document