scholarly journals Progressively Inpainting Images Based on a Forked-Then-Fused Decoder Network

Sensors ◽  
2021 ◽  
Vol 21 (19) ◽  
pp. 6336
Author(s):  
Shuai Yang ◽  
Rong Huang ◽  
Fang Han
Keyword(s):  
Long Range ◽  
State Of The Art ◽  
Image Inpainting ◽  
Basic Component ◽  
Qualitative And Quantitative ◽  
Multi Scale ◽  
Art Works ◽  
Quantitative Results

Image inpainting aims to fill in corrupted regions with visually realistic and semantically plausible contents. In this paper, we propose a progressive image inpainting method, which is based on a forked-then-fused decoder network. A unit called PC-RN, which is the combination of partial convolution and region normalization, serves as the basic component to construct inpainting network. The PC-RN unit can extract useful features from the valid surroundings and can suppress incompleteness-caused interference at the same time. The forked-then-fused decoder network consists of a local reception branch, a long-range attention branch, and a squeeze-and-excitation-based fusing module. Two multi-scale contextual attention modules are deployed into the long-range attention branch for adaptively borrowing features from distant spatial positions. Progressive inpainting strategy allows the attention modules to use the previously filled region to reduce the risk of allocating wrong attention. We conduct extensive experiments on three benchmark databases: Places2, Paris StreetView, and CelebA. Qualitative and quantitative results show that the proposed inpainting model is superior to state-of-the-art works. Moreover, we perform ablation studies to reveal the functionality of each module for the image inpainting task.

scholarly journals FusionDN: A Unified Densely Connected Network for Image Fusion

2020 ◽  
Vol 34 (07) ◽  
pp. 12484-12491 ◽  
Author(s):  
Han Xu ◽  
Jiayi Ma ◽  
Zhuliang Le ◽  
Junjun Jiang ◽  
Xiaojie Guo
Keyword(s):  
Image Fusion ◽  
State Of The Art ◽  
Data Driven ◽  
Single Model ◽  
Connected Network ◽  
Qualitative And Quantitative ◽  
Multiple Tasks ◽  
Different Types ◽  
Fused Image ◽  
Quantitative Results

In this paper, we present a new unsupervised and unified densely connected network for different types of image fusion tasks, termed as FusionDN. In our method, the densely connected network is trained to generate the fused image conditioned on source images. Meanwhile, a weight block is applied to obtain two data-driven weights as the retention degrees of features in different source images, which are the measurement of the quality and the amount of information in them. Losses of similarities based on these weights are applied for unsupervised learning. In addition, we obtain a single model applicable to multiple fusion tasks by applying elastic weight consolidation to avoid forgetting what has been learned from previous tasks when training multiple tasks sequentially, rather than train individual models for every fusion task or jointly train tasks roughly. Qualitative and quantitative results demonstrate the advantages of FusionDN compared with state-of-the-art methods in different fusion tasks.

scholarly journals Reversible Privacy Protection with the Capability of Antiforensics

2021 ◽  
Vol 2021 ◽  
pp. 1-12
Author(s):  
Liyun Dou ◽  
Zichi Wang ◽  
Zhenxing Qian ◽  
Guorui Feng
Keyword(s):  
Data Hiding ◽  
Privacy Protection ◽  
State Of The Art ◽  
Image Inpainting ◽  
Single Image ◽  
Interactive Approach ◽  
Protection Scheme ◽  
Objective Metrics ◽  
Art Works ◽  
Pattern Noise

In this paper, we propose a privacy protection scheme using image dual-inpainting and data hiding. In the proposed scheme, the privacy contents in the original image are concealed, which are reversible that the privacy content can be perfectly recovered. We use an interactive approach to select the areas to be protected, that is, the protection data. To address the disadvantage that single image inpainting is susceptible to forensic localization, we propose a dual-inpainting algorithm to implement the object removal task. The protection data is embedded into the image with object removed using a popular data hiding method. We further use the pattern noise forensic detection and the objective metrics to assess the proposed method. The results on different scenarios show that the proposed scheme can achieve better visual quality and antiforensic capability than the state-of-the-art works.

scholarly journals Mitigating Demographic Bias in Facial Datasets with Style-Based Multi-attribute Transfer

2021 ◽  
Author(s):  
Markos Georgopoulos ◽  
James Oldfield ◽  
Mihalis A. Nicolaou ◽  
Yannis Panagakis ◽  
Maja Pantic
Keyword(s):  
Deep Learning ◽  
Data Augmentation ◽  
State Of The Art ◽  
Skin Tone ◽  
Training Set ◽  
Demographic Groups ◽  
Underrepresented Groups ◽  
Qualitative And Quantitative ◽  
Multi Scale ◽  
Single Attribute

AbstractDeep learning has catalysed progress in tasks such as face recognition and analysis, leading to a quick integration of technological solutions in multiple layers of our society. While such systems have proven to be accurate by standard evaluation metrics and benchmarks, a surge of work has recently exposed the demographic bias that such algorithms exhibit–highlighting that accuracy does not entail fairness. Clearly, deploying biased systems under real-world settings can have grave consequences for affected populations. Indeed, learning methods are prone to inheriting, or even amplifying the bias present in a training set, manifested by uneven representation across demographic groups. In facial datasets, this particularly relates to attributes such as skin tone, gender, and age. In this work, we address the problem of mitigating bias in facial datasets by data augmentation. We propose a multi-attribute framework that can successfully transfer complex, multi-scale facial patterns even if these belong to underrepresented groups in the training set. This is achieved by relaxing the rigid dependence on a single attribute label, and further introducing a tensor-based mixing structure that captures multiplicative interactions between attributes in a multilinear fashion. We evaluate our method with an extensive set of qualitative and quantitative experiments on several datasets, with rigorous comparisons to state-of-the-art methods. We find that the proposed framework can successfully mitigate dataset bias, as evinced by extensive evaluations on established diversity metrics, while significantly improving fairness metrics such as equality of opportunity.

scholarly journals Human Synthesis and Scene Compositing

2020 ◽  
Vol 34 (07) ◽  
pp. 12749-12756 ◽  
Author(s):  
Mihai Zanfir ◽  
Elisabeta Oneata ◽  
Alin-Ionut Popa ◽  
Andrei Zanfir ◽  
Cristian Sminchisescu
Keyword(s):  
State Of The Art ◽  
Image Synthesis ◽  
Parametric Representation ◽  
Special Effects ◽  
Qualitative And Quantitative ◽  
Quantitative Results ◽  
Visual Artifacts ◽  
Photo Editing ◽  
Human Image ◽  
3D Scene

Generating good quality and geometrically plausible synthetic images of humans with the ability to control appearance, pose and shape parameters, has become increasingly important for a variety of tasks ranging from photo editing, fashion virtual try-on, to special effects and image compression. In this paper, we propose a HUSC (HUman Synthesis and Scene Compositing) framework for the realistic synthesis of humans with different appearance, in novel poses and scenes. Central to our formulation is 3d reasoning for both people and scenes, in order to produce realistic collages, by correctly modeling perspective effects and occlusion, by taking into account scene semantics and by adequately handling relative scales. Conceptually our framework consists of three components: (1) a human image synthesis model with controllable pose and appearance, based on a parametric representation, (2) a person insertion procedure that leverages the geometry and semantics of the 3d scene, and (3) an appearance compositing process to create a seamless blending between the colors of the scene and the generated human image, and avoid visual artifacts. The performance of our framework is supported by both qualitative and quantitative results, in particular state-of-the art synthesis scores for the DeepFashion dataset.

scholarly journals Random-Shaped Image Inpainting using Dilated Convolution

2019 ◽  
Vol 8 (6) ◽  
pp. 641-647
Keyword(s):  
Deep Learning ◽  
State Of The Art ◽  
Image Inpainting ◽  
Mean Value ◽  
Variable Size ◽  
Qualitative And Quantitative ◽  
The Past ◽  
New Novel ◽  
The Mean ◽  
Image Pixels

Over the past few years, Deep learning-based methods have shown encouraging and inspiring results for one of the most complex tasks of computer vision and image processing; Image Inpainting. The difficulty of image inpainting is derived from its’ need to fully and deeply understand of the structure and texture of images for producing accurate and visibly plausible results especially for the cases of inpainting a relatively larger region. Deep learning methods usually employ convolution neural network (CNN) for processing and analyzing images using filters that consider all image pixels as valid ones and usually use the mean value to substitute the missing pixels. This result in artifacts and blurry inpainted regions inconsistent with the rest of the image. In this paper, a new novel-based method is proposed for image inpainting of random-shaped missing regions with variable size and arbitrary locations across the image. We employed the use of dilated convolutions for composing multiscale context information without any loss in resolution as well as including a modification mask step after each convolution operation. The proposed method also includes a global discriminator that also considers the scale of patches as well as the whole image. The global discriminator is responsible for capturing local continuity of images texture as well as the overall global images’ features. The performance of the proposed method is evaluated using two datasets (Places2 and Paris Street View). Also, a comparison with the recent state-of-the-art is preformed to demonstrate and prove the effectiveness of our model in both qualitative and quantitative evaluations.

scholarly journals A Cross-Direction and Progressive Network for Pan-Sharpening

2021 ◽  
Vol 13 (15) ◽  
pp. 3045
Author(s):  
Han Xu ◽  
Zhuliang Le ◽  
Jun Huang ◽  
Jiayi Ma
Keyword(s):  
State Of The Art ◽  
Ground Truth ◽  
Single Type ◽  
Spectral Information ◽  
Cross Direction ◽  
Qualitative And Quantitative ◽  
Multi Scale ◽  
Fused Image ◽  
Partial Inactivation ◽  
The One

In this paper, we propose a cross-direction and progressive network, termed CPNet, to solve the pan-sharpening problem. The full processing of information is the main characteristic of our model, which is reflected as follows: on the one hand, we process the source images in a cross-direction manner to obtain the source images of different scales as the input of the fusion modules at different stages, which maximizes the usage of multi-scale information in the source images; on the other hand, the progressive reconstruction loss is designed to boost the training of our network and avoid partial inactivation, while maintaining the consistency of the fused result with the ground truth. Since the extraction of the information from the source images and the reconstruction of the fused image is based on the entire image rather than a single type of information, there is little loss of partial spatial or spectral information due to insufficient information processing. Extensive experiments, including qualitative and quantitative comparisons demonstrate that our model can maintain more spatial and spectral information compared to the state-of-the-art pan-sharpening methods.

scholarly journals A Sketch-Transformer Network for Face Photo-Sketch Synthesis

2021 ◽  
Author(s):  
Mingrui Zhu ◽  
Changcheng Liang ◽  
Nannan Wang ◽  
Xiaoyu Wang ◽  
Zhifeng Li ◽  
...  
Keyword(s):  
Long Range ◽  
State Of The Art ◽  
Structural Information ◽  
Generative Adversarial Networks ◽  
Spatial Dependency ◽  
Facial Image ◽  
Multi Scale ◽  
Adversarial Networks ◽  
Spatially Adaptive ◽  
Sketch Synthesis

We present a face photo-sketch synthesis model, which converts a face photo into an artistic face sketch or recover a photo-realistic facial image from a sketch portrait. Recent progress has been made by convolutional neural networks (CNNs) and generative adversarial networks (GANs), so that promising results can be obtained through real-time end-to-end architectures. However, convolutional architectures tend to focus on local information and neglect long-range spatial dependency, which limits the ability of existing approaches in keeping global structural information. In this paper, we propose a Sketch-Transformer network for face photo-sketch synthesis, which consists of three closely-related modules, including a multi-scale feature and position encoder for patch-level feature and position embedding, a self-attention module for capturing long-range spatial dependency, and a multi-scale spatially-adaptive de-normalization decoder for image reconstruction. Such a design enables the model to generate reasonable detail texture while maintaining global structural information. Extensive experiments show that the proposed method achieves significant improvements over state-of-the-art approaches on both quantitative and qualitative evaluations.

scholarly journals Precipitation Modeling for Extreme Weather Based on Sparse Hybrid Machine Learning and Markov Chain Random Field in a Multi-Scale Subspace

Water ◽  
2021 ◽  
Vol 13 (9) ◽  
pp. 1241
Author(s):  
Ming-Hsi Lee ◽  
Yenming J. Chen
Keyword(s):  
Machine Learning ◽  
Markov Chain ◽  
Random Field ◽  
Long Range ◽  
Weather Conditions ◽  
Extreme Weather ◽  
Prediction Algorithm ◽  
Learning Method ◽  
Multi Scale ◽  
Hybrid Machine

This paper proposes to apply a Markov chain random field conditioning method with a hybrid machine learning method to provide long-range precipitation predictions under increasingly extreme weather conditions. Existing precipitation models are limited in time-span, and long-range simulations cannot predict rainfall distribution for a specific year. This paper proposes a hybrid (ensemble) learning method to perform forecasting on a multi-scaled, conditioned functional time series over a sparse l1 space. Therefore, on the basis of this method, a long-range prediction algorithm is developed for applications, such as agriculture or construction works. Our findings show that the conditioning method and multi-scale decomposition in the parse space l1 are proved useful in resisting statistical variation due to increasingly extreme weather conditions. Because the predictions are year-specific, we verify our prediction accuracy for the year we are interested in, but not for other years.

Sign in / Sign up

Export Citation Format

Share Document