Divide-and-Conquer Federated Learning Under Data Heterogeneity

Federated Learning allows training of data stored in distributed devices without the need for centralizing training-data, thereby maintaining data-privacy. Addressing the ability to handle data heterogeneity (non-identical and independent distribution or non-IID) is a key enabler for the wider deployment of Federated Learning. In this paper, we propose a novel Divide-andConquer training methodology that enables the use of the popular FedAvg aggregation algorithm by over-coming the acknowledged FedAvg limitations in non-IID environments. We propose a novel use of Cosine-distance based Weight Divergence metric to determine the exact point where a Deep Learning network can be divided into class-agnostic initial layers and class-specific deep layers for performing a Divide and Conquer training. We show that the methodology achieves trained-model accuracy at-par with (and in certain cases exceeding) the numbers achieved by state-of-the-art algorithms like FedProx, FedMA, etc. Also, we show that this methodology leads to compute and/or bandwidth optimizations under certain documented conditions.

Download Full-text

2D–3D reconstruction of distal forearm bone from actual X-ray images of the wrist using convolutional neural networks

Scientific Reports ◽

10.1038/s41598-021-94634-2 ◽

2021 ◽

Vol 11 (1) ◽

Author(s):

Ryoya Shiode ◽

Mototaka Kabashima ◽

Yuta Hiasa ◽

Kunihiro Oka ◽

Tsuyoshi Murase ◽

...

Keyword(s):

Wrist Joint ◽

High Accuracy ◽

Training Data ◽

Small Data ◽

Data Set ◽

Accuracy Estimation ◽

X Ray ◽

Learning Network ◽

Forearm Bone ◽

Deep Learning Network

AbstractThe purpose of the study was to develop a deep learning network for estimating and constructing highly accurate 3D bone models directly from actual X-ray images and to verify its accuracy. The data used were 173 computed tomography (CT) images and 105 actual X-ray images of a healthy wrist joint. To compensate for the small size of the dataset, digitally reconstructed radiography (DRR) images generated from CT were used as training data instead of actual X-ray images. The DRR-like images were generated from actual X-ray images in the test and adapted to the network, and high-accuracy estimation of a 3D bone model from a small data set was possible. The 3D shape of the radius and ulna were estimated from actual X-ray images with accuracies of 1.05 ± 0.36 and 1.45 ± 0.41 mm, respectively.

Download Full-text

Performance Evaluation of Deep CNN-Based Crack Detection and Localization Techniques for Concrete Structures

Sensors ◽

10.3390/s21051688 ◽

2021 ◽

Vol 21 (5) ◽

pp. 1688

Author(s):

Luqman Ali ◽

Fady Alnajjar ◽

Hamad Al Jassmi ◽

Munkhjargal Gochoo ◽

Wasif Khan ◽

...

Keyword(s):

Neural Network ◽

Convolutional Neural Network ◽

Crack Detection ◽

Concrete Structures ◽

Model Performance ◽

Training Data ◽

Computational Time ◽

Data Heterogeneity ◽

Public Datasets ◽

Detection And Localization

This paper proposes a customized convolutional neural network for crack detection in concrete structures. The proposed method is compared to four existing deep learning methods based on training data size, data heterogeneity, network complexity, and the number of epochs. The performance of the proposed convolutional neural network (CNN) model is evaluated and compared to pretrained networks, i.e., the VGG-16, VGG-19, ResNet-50, and Inception V3 models, on eight datasets of different sizes, created from two public datasets. For each model, the evaluation considered computational time, crack localization results, and classification measures, e.g., accuracy, precision, recall, and F1-score. Experimental results demonstrated that training data size and heterogeneity among data samples significantly affect model performance. All models demonstrated promising performance on a limited number of diverse training data; however, increasing the training data size and reducing diversity reduced generalization performance, and led to overfitting. The proposed customized CNN and VGG-16 models outperformed the other methods in terms of classification, localization, and computational time on a small amount of data, and the results indicate that these two models demonstrate superior crack detection and localization for concrete structures.

Download Full-text

Monte Carlo and Reconstruction Membership Inference Attacks against Generative Models

Proceedings on Privacy Enhancing Technologies ◽

10.2478/popets-2019-0067 ◽

2019 ◽

Vol 2019 (4) ◽

pp. 232-249 ◽

Cited By ~ 1

Author(s):

Benjamin Hilprecht ◽

Martin Härterich ◽

Daniel Bernau

Keyword(s):

Data Privacy ◽

Information Leakage ◽

Generative Models ◽

Generative Model ◽

Training Data ◽

Generative Adversarial Networks ◽

Data Sets ◽

Success Rates ◽

Model Quality ◽

Type Formalization

Abstract We present two information leakage attacks that outperform previous work on membership inference against generative models. The first attack allows membership inference without assumptions on the type of the generative model. Contrary to previous evaluation metrics for generative models, like Kernel Density Estimation, it only considers samples of the model which are close to training data records. The second attack specifically targets Variational Autoencoders, achieving high membership inference accuracy. Furthermore, previous work mostly considers membership inference adversaries who perform single record membership inference. We argue for considering regulatory actors who perform set membership inference to identify the use of specific datasets for training. The attacks are evaluated on two generative model architectures, Generative Adversarial Networks (GANs) and Variational Autoen-coders (VAEs), trained on standard image datasets. Our results show that the two attacks yield success rates superior to previous work on most data sets while at the same time having only very mild assumptions. We envision the two attacks in combination with the membership inference attack type formalization as especially useful. For example, to enforce data privacy standards and automatically assessing model quality in machine learning as a service setups. In practice, our work motivates the use of GANs since they prove less vulnerable against information leakage attacks while producing detailed samples.

Download Full-text

Accurate refinement of docked protein complexes using evolutionary information and deep learning

Journal of Bioinformatics and Computational Biology ◽

10.1142/s0219720016420026 ◽

2016 ◽

Vol 14 (03) ◽

pp. 1642002 ◽

Cited By ~ 11

Author(s):

Bahar Akbal-Delibas ◽

Roshanak Farhoodi ◽

Marc Pomplun ◽

Nurit Haspel

Keyword(s):

Deep Learning ◽

Protein Complexes ◽

Scoring Function ◽

Protein Docking ◽

Training Data ◽

Evolutionary Information ◽

Native Structure ◽

Learning Network ◽

Small Set ◽

Deep Learning Network

One of the major challenges for protein docking methods is to accurately discriminate native-like structures from false positives. Docking methods are often inaccurate and the results have to be refined and re-ranked to obtain native-like complexes and remove outliers. In a previous work, we introduced AccuRefiner, a machine learning based tool for refining protein–protein complexes. Given a docked complex, the refinement tool produces a small set of refined versions of the input complex, with lower root-mean-square-deviation (RMSD) of atomic positions with respect to the native structure. The method employs a unique ranking tool that accurately predicts the RMSD of docked complexes with respect to the native structure. In this work, we use a deep learning network with a similar set of features and five layers. We show that a properly trained deep learning network can accurately predict the RMSD of a docked complex with 1.40 Å error margin on average, by approximating the complex relationship between a wide set of scoring function terms and the RMSD of a docked structure. The network was trained on 35000 unbound docking complexes generated by RosettaDock. We tested our method on 25 different putative docked complexes produced also by RosettaDock for five proteins that were not included in the training data. The results demonstrate that the high accuracy of the ranking tool enables AccuRefiner to consistently choose the refinement candidates with lower RMSD values compared to the coarsely docked input structures.

Download Full-text

A Computationally Efficient Methodology for Generating Training Data for a Transient Neural Network of a Tip-Jet Reaction Drive System

Journal of Engineering for Gas Turbines and Power ◽

10.1115/1.4003957 ◽

2011 ◽

Vol 133 (12) ◽

Cited By ~ 6

Author(s):

Brian K. Kestner ◽

Jimmy C.M. Tai ◽

Dimitri N. Mavris

Keyword(s):

Neural Network ◽

Steady State ◽

Network Model ◽

Neural Network Model ◽

Operating Conditions ◽

Drive System ◽

Training Data ◽

Computationally Efficient ◽

Model Based Control ◽

Training Methodology

This paper presents a computationally efficient methodology for generating training data for a transient neural network model of a tip-jet reaction drive system for potential use as an onboard model in a model based control application. This methodology significantly reduces the number of training points required to capture the transient performance of the system. The challenge in developing an onboard model for a tip-jet reaction drive system is that the model has to operate over the whole flight envelope, to account for the different dynamics present in the system, and to adjust to system degradation or potential faults. In addition, the onboard model must execute in less time than the update interval of the controller. To address these issues, a computationally efficient training methodology and neural network surrogate model have been developed that captures the transient performance of the tip-jet reaction system. As the number of inputs to a neural network becomes large, the computational time needed to generate the number of training points required to accurately represent the range of operating conditions of the system may become quite large also. A challenge for the tip-jet reaction drive system is to minimize the number of neural network training points, while maintaining the high accuracy. To address this issue, a novel training methodology is presented which first trains a steady-state neural network model and uses deviations from steady-state operating conditions to define the transient portion of the training data. The combined results from both the transient and the steady-state training data can then be used to create a single transient neural network of the system. The results in this paper demonstrate that a transient neural network using this new computationally efficient training methodology has the potential to be a feasible option for use as an onboard real-time model for model based control of a tip-jet reaction drive system.

Download Full-text

R-CRISPR: A Deep Learning Network to Predict Off-Target Activities with Mismatch, Insertion and Deletion in CRISPR-Cas9 System

Genes ◽

10.3390/genes12121878 ◽

2021 ◽

Vol 12 (12) ◽

pp. 1878

Author(s):

Rui Niu ◽

Jiajie Peng ◽

Zhipeng Zhang ◽

Xuequn Shang

Keyword(s):

Neural Network ◽

Deep Learning ◽

In Silico ◽

Prediction Method ◽

Training Data ◽

Prediction Methods ◽

In Silico Prediction ◽

Guide Rnas ◽

Insertion And Deletion ◽

Deep Learning Network

The Clustered Regularly Interspaced Short Palindromic Repeats (CRISPR)—associated protein 9 (Cas9) system is a groundbreaking gene-editing tool, which has been widely adopted in biomedical research. However, the guide RNAs in CRISPR-Cas9 system may induce unwanted off-target activities and further affect the practical application of the technique. Most existing in silico prediction methods that focused on off-target activities possess limited predictive precision and remain to be improved. Hence, it is necessary to propose a new in silico prediction method to address this problem. In this work, a deep learning framework named R-CRISPR is presented, which devises an encoding scheme to encode gRNA-target sequences into binary matrices, a convolutional neural network as feature extractor, and a recurrent neural network to predict off-target activities with mismatch, insertion, or deletion. It is demonstrated that R-CRISPR surpasses six mainstream prediction methods with a significant improvement on mismatch-only datasets verified by GUIDE-seq. Compared with the state-of-art prediction methods, R-CRISPR also achieves competitive performance on datasets with mismatch, insertion, and deletion. Furthermore, experiments show that data concatenate could influence the quality of training data, and investigate the optimal combination of datasets.

Download Full-text

Few-shot Classification of Aerial Scene Images via Meta-learning

10.20944/preprints202010.0033.v1 ◽

2020 ◽

Cited By ~ 1

Author(s):

Pei Zhang ◽

YIng Li ◽

Dong Wang ◽

Yunpeng Bai

Keyword(s):

Classification Problem ◽

Training Data ◽

Scene Classification ◽

Shot Classification ◽

Training Stage ◽

Testing Stage ◽

Meta Learning ◽

Cosine Distance ◽

Query Sample

CNN-based methods have dominated the field of aerial scene classification for the past few years. While achieving remarkable success, CNN-based methods suffer from excessive parameters and notoriously rely on large amounts of training data. In this work, we introduce few-shot learning to the aerial scene classification problem. Few-shot learning aims to learn a model on base-set that can quickly adapt to unseen categories in novel-set, using only a few labeled samples. To this end, we proposed a meta-learning method for few-shot classification of aerial scene images. First, we train a feature extractor on all base categories to learn a representation of inputs. Then in the meta-training stage, the classifier is optimized in the metric space by cosine distance with a learnable scale parameter. At last, in the meta-testing stage, the query sample in the unseen category is predicted by the adapted classifier given a few support samples. We conduct extensive experiments on two challenging datasets: NWPU-RESISC45 and RSD46-WHU. The experimental results show that our method outperforms three state-of-the-art few-shot algorithms and one typical CNN-based method, D-CNN. Furthermore, several ablation experiments are conducted to investigate the effects of dataset scale and support shots; the experiment results confirm that our model is specifically effective in few-shot settings.

Download Full-text

Towards Understanding the Risks of Gradient Inversion in Federated Learning

10.21203/rs.3.rs-1147182/v2 ◽

2021 ◽

Author(s):

Ali Hatamizadeh ◽

Hongxu Yin ◽

Pavlo Molchanov ◽

Andriy Myronenko ◽

Wenqi Li ◽

...

Keyword(s):

Neural Networks ◽

Data Privacy ◽

Deep Neural Networks ◽

Differential Privacy ◽

Use Cases ◽

Training Data ◽

Model Accuracy ◽

Healthcare Applications ◽

Raw Data ◽

Collaborative Training

Abstract Federated learning (FL) allows the collaborative training of AI models without needing to share raw data. This capability makes it especially interesting for healthcare applications where patient and data privacy is of utmost concern. However, recent works on the inversion of deep neural networks from model gradients raised concerns about the security of FL in preventing the leakage of training data. In this work, we show that these attacks presented in the literature are impractical in real FL use-cases and provide a new baseline attack that works for more realistic scenarios where the clients’ training involves updating the Batch Normalization (BN) statistics. Furthermore, we present new ways to measure and visualize potential data leakage in FL. Our work is a step towards establishing reproducible methods of measuring data leakage in FL and could help determine the optimal tradeoffs between privacy-preserving techniques, such as differential privacy, and model accuracy based on quantifiable metrics.

Download Full-text

Data Augmentation Method by Applying Color Perturbation of Inverse PSNR and Geometric Transformations for Object Recognition Based on Deep Learning

Applied Sciences ◽

10.3390/app10113755 ◽

2020 ◽

Vol 10 (11) ◽

pp. 3755

Author(s):

Eun Kyeong Kim ◽

Hansoo Lee ◽

Jin Yong Kim ◽

Sungshin Kim

Keyword(s):

Deep Learning ◽

Object Recognition ◽

Data Augmentation ◽

Color Space ◽

Original Data ◽

Training Data ◽

Geometric Transformation ◽

Geometric Transformations ◽

Background Elimination ◽

Deep Learning Network

Deep learning is applied in various manufacturing domains. To train a deep learning network, we must collect a sufficient amount of training data. However, it is difficult to collect image datasets required to train the networks to perform object recognition, especially because target items that are to be classified are generally excluded from existing databases, and the manual collection of images poses certain limitations. Therefore, to overcome the data deficiency that is present in many domains including manufacturing, we propose a method of generating new training images via image pre-processing steps, background elimination, target extraction while maintaining the ratio of the object size in the original image, color perturbation considering the predefined similarity between the original and generated images, geometric transformations, and transfer learning. Specifically, to demonstrate color perturbation and geometric transformations, we compare and analyze the experiments of each color space and each geometric transformation. The experimental results show that the proposed method can effectively augment the original data, correctly classify similar items, and improve the image classification accuracy. In addition, it also demonstrates that the effective data augmentation method is crucial when the amount of training data is small.

Download Full-text

Public Data Integrity Verification Scheme for Secure Cloud Storage

Information ◽

10.3390/info11090409 ◽

2020 ◽

Vol 11 (9) ◽

pp. 409

Author(s):

Yuan Ping ◽

Yu Zhan ◽

Ke Lu ◽

Baocang Wang

Keyword(s):

Cloud Storage ◽

Data Privacy ◽

Data Integrity ◽

Third Party ◽

Divide And Conquer ◽

Data Outsourcing ◽

Verification Scheme ◽

Integrity Verification ◽

Outsourced Data ◽

Public Data

Although cloud storage provides convenient data outsourcing services, an untrusted cloud server frequently threatens the integrity and security of the outsourced data. Therefore, it is extremely urgent to design security schemes allowing the users to check the integrity of data with acceptable computational and communication overheads. In this paper, we first propose a public data integrity verification scheme based on the algebraic signature and elliptic curve cryptography. This scheme not only allows the third party authority deputize for users to verify the outsourced data integrity, but also resists malicious attacks such as replay attacks, replacing attack and forgery attacks. Data privacy is guaranteed by symmetric encryption. Furthermore, we construct a novel data structure named divide and conquer hash list, which can efficiently perform data updating operations, such as deletion, insertion, and modification. Compared with the relevant schemes in the literature, security analysis and performance evaluations show that the proposed scheme gains some advantages in integrity verification and dynamic updating.

Download Full-text