scholarly journals A temporal-spectral generative adversarial network based end-to-end packet loss concealment for wideband speech transmission

2021 ◽  
Vol 150 (4) ◽  
pp. 2577-2588
Author(s):  
Jie Wang ◽  
Yuansheng Guan ◽  
Chengshi Zheng ◽  
Renhua Peng ◽  
Xiaodong Li
Keyword(s):  
Packet Loss ◽  
Generative Adversarial Network ◽  
Adversarial Network ◽  
Speech Transmission ◽  
End To End

scholarly journals EnsNet: Ensconce Text in the Wild

2019 ◽  
Vol 33 ◽  
pp. 801-808 ◽  
Author(s):  
Shuaitao Zhang ◽  
Yuliang Liu ◽  
Lianwen Jin ◽  
Yaoxiong Huang ◽  
Songxuan Lai
Keyword(s):  
Generative Adversarial Network ◽  
Local Consistency ◽  
Adversarial Network ◽  
Image Patches ◽  
Scene Text ◽  
In The Wild ◽  
Lateral Connection ◽  
Previous State ◽  
End To End ◽  
General Object

A new method is proposed for removing text from natural images. The challenge is to first accurately localize text on the stroke-level and then replace it with a visually plausible background. Unlike previous methods that require image patches to erase scene text, our method, namely ensconce network (EnsNet), can operate end-to-end on a single image without any prior knowledge. The overall structure is an end-to-end trainable FCN-ResNet-18 network with a conditional generative adversarial network (cGAN). The feature of the former is first enhanced by a novel lateral connection structure and then refined by four carefully designed losses: multiscale regression loss and content loss, which capture the global discrepancy of different level features; texture loss and total variation loss, which primarily target filling the text region and preserving the reality of the background. The latter is a novel local-sensitive GAN, which attentively assesses the local consistency of the text erased regions. Both qualitative and quantitative sensitivity experiments on synthetic images and the ICDAR 2013 dataset demonstrate that each component of the EnsNet is essential to achieve a good performance. Moreover, our EnsNet can significantly outperform previous state-of-the-art methods in terms of all metrics. In addition, a qualitative experiment conducted on the SBMNet dataset further demonstrates that the proposed method can also preform well on general object (such as pedestrians) removal tasks. EnsNet is extremely fast, which can preform at 333 fps on an i5-8600 CPU device.

scholarly journals End-to-End Unpaired Image Denoising with Conditional Adversarial Networks

2020 ◽  
Vol 34 (04) ◽  
pp. 4140-4149
Author(s):  
Zhiwei Hong ◽  
Xiaocheng Fan ◽  
Tao Jiang ◽  
Jianxing Feng
Keyword(s):  
Image Denoising ◽  
Real Data ◽  
Training Data ◽  
Noise Distribution ◽  
Data Types ◽  
Generative Adversarial Network ◽  
Noisy Images ◽  
Adversarial Network ◽  
Noisy Observation ◽  
End To End

Image denoising is a classic low level vision problem that attempts to recover a noise-free image from a noisy observation. Recent advances in deep neural networks have outperformed traditional prior based methods for image denoising. However, the existing methods either require paired noisy and clean images for training or impose certain assumptions on the noise distribution and data types. In this paper, we present an end-to-end unpaired image denoising framework (UIDNet) that denoises images with only unpaired clean and noisy training images. The critical component of our model is a noise learning module based on a conditional Generative Adversarial Network (cGAN). The model learns the noise distribution from the input noisy images and uses it to transform the input clean images to noisy ones without any assumption on the noise distribution and data types. This process results in pairs of clean and pseudo-noisy images. Such pairs are then used to train another denoising network similar to the existing denoising methods based on paired images. The noise learning and denoising components are integrated together so that they can be trained end-to-end. Extensive experimental evaluation has been performed on both synthetic and real data including real photographs and computer tomography (CT) images. The results demonstrate that our model outperforms the previous models trained on unpaired images as well as the state-of-the-art methods based on paired training data when proper training pairs are unavailable.

scholarly journals Adding Crowd Noise to Sports Commentary using Generative Models

2021 ◽  
Author(s):  
Neil Shah ◽  
Dharmeshkumar M. Agrawal ◽  
Niranajan Pedanekar
Keyword(s):  
Time Domain ◽  
Source Separation ◽  
Generative Models ◽  
User Evaluation ◽  
Generative Adversarial Network ◽  
Adversarial Network ◽  
End To End ◽  
Sports Commentary ◽  
Domain Source

Crowd noise forms an integral part of a live sports experience. In the post-COVID era, when live audiences are absent, crowd noise needs to be added to the live commentary. This paper exploits the correlation between commentary and crowd noise of a live sports event and presents an audio stylizing sports commentary method by generating live stadium-like sound using neural generative models. We use the Generative Adversarial Network (GAN)-based architectures such as Cycle-consistent GANs (Cycle-GANs) and Mel-GANs to generate live stadium-like sound samples given the live commentary. Due to the unavailability of raw commentary sound samples, we use end-to-end time-domain source separation models (SEGAN and Wave-U-Net) to extract commentary sound from combined recordings of the live sound acquired from YouTube highlights of soccer videos. We present a qualitative and a subjective user evaluation of the similarity of the generated live sound with the reference live sound.

scholarly journals Speech-driven Facial Animations Improve Speech-in-Noise Comprehension of Humans

2021 ◽  
Author(s):  
Enrico Varano ◽  
Konstantinos Vougioukas ◽  
Pingchuan Ma ◽  
Stavros Petridis ◽  
Maja Pantic ◽  
...  
Keyword(s):  
Hearing Impairment ◽  
Audiovisual Speech ◽  
Speech Comprehension ◽  
Generative Adversarial Network ◽  
Still Image ◽  
Speech In Noise ◽  
Adversarial Network ◽  
Listening Environments ◽  
End To End ◽  
Speech Recognizer

Understanding speech becomes a demanding task when the environment is noisy. Comprehension of speech in noise can be substantially improved by looking at the speake's face, and this audiovisual benefit is even more pronounced in people with hearing impairment. Recent advances in AI have allowed to synthesize photorealistic talking faces from a speech recording and a still image of a person's face in an end-to-end manner. However, it has remained unknown whether such facial animations improve speech-in-noise comprehension. Here we consider facial animations produced by a recently introduced generative adversarial network (GAN), and show that humans cannot distinguish between the synthesized and the natural videos. Importantly, we then show that the end-to-end synthesized videos significantly aid humans in understanding speech in noise, although the natural facial motions yield a yet higher audiovisual benefit. We further find that an audiovisual speech recognizer benefits from the synthesized facial animations as well. Our results suggest that synthesizing facial motions from speech can be used to aid speech comprehension in difficult listening environments.

Sign in / Sign up

Export Citation Format

Share Document