scholarly journals H-FL: A Hierarchical Communication-Efficient and Privacy-Protected Architecture for Federated Learning

2021 ◽  
Author(s):  
He Yang
Keyword(s):  
Differential Privacy ◽  
Model Performance ◽  
Training Data ◽  
Communication Overhead ◽  
Complete Model ◽  
Model Accuracy ◽  
Moderate Amount ◽  
Reconstruction Strategy ◽  
Hierarchical Communication ◽  
World Image

The longstanding goals of federated learning (FL) require rigorous privacy guarantees and low communication overhead while holding a relatively high model accuracy. However, simultaneously achieving all the goals is extremely challenging. In this paper, we propose a novel framework called hierarchical federated learning (H-FL) to tackle this challenge. Considering the degradation of the model performance due to the statistic heterogeneity of the training data, we devise a runtime distribution reconstruction strategy, which reallocates the clients appropriately and utilizes mediators to rearrange the local training of the clients. In addition, we design a compression-correction mechanism incorporated into H-FL to reduce the communication overhead while not sacrificing the model performance. To further provide privacy guarantees, we introduce differential privacy while performing local training, which injects moderate amount of noise into only part of the complete model. Experimental results show that our H-FL framework achieves the state-of-art performance on different datasets for the real-world image recognition tasks.

scholarly journals Privacy-Preserving Gradient Boosting Decision Trees

2020 ◽  
Vol 34 (01) ◽  
pp. 784-791 ◽  
Author(s):  
Qinbin Li ◽  
Zhaomin Wu ◽  
Zeyi Wen ◽  
Bingsheng He
Keyword(s):  
Machine Learning ◽  
Differential Privacy ◽  
Training Data ◽  
Gradient Boosting ◽  
Training Algorithm ◽  
Model Accuracy ◽  
Machine Learning Model ◽  
Improve Model ◽  
Privacy Budget ◽  
Privacy Level

The Gradient Boosting Decision Tree (GBDT) is a popular machine learning model for various tasks in recent years. In this paper, we study how to improve model accuracy of GBDT while preserving the strong guarantee of differential privacy. Sensitivity and privacy budget are two key design aspects for the effectiveness of differential private models. Existing solutions for GBDT with differential privacy suffer from the significant accuracy loss due to too loose sensitivity bounds and ineffective privacy budget allocations (especially across different trees in the GBDT model). Loose sensitivity bounds lead to more noise to obtain a fixed privacy level. Ineffective privacy budget allocations worsen the accuracy loss especially when the number of trees is large. Therefore, we propose a new GBDT training algorithm that achieves tighter sensitivity bounds and more effective noise allocations. Specifically, by investigating the property of gradient and the contribution of each tree in GBDTs, we propose to adaptively control the gradients of training data for each iteration and leaf node clipping in order to tighten the sensitivity bounds. Furthermore, we design a novel boosting framework to allocate the privacy budget between trees so that the accuracy loss can be further reduced. Our experiments show that our approach can achieve much better model accuracy than other baselines.

scholarly journals Towards Understanding the Risks of Gradient Inversion in Federated Learning

2021 ◽  
Author(s):  
Ali Hatamizadeh ◽  
Hongxu Yin ◽  
Pavlo Molchanov ◽  
Andriy Myronenko ◽  
Wenqi Li ◽  
...  
Keyword(s):  
Neural Networks ◽  
Data Privacy ◽  
Deep Neural Networks ◽  
Differential Privacy ◽  
Use Cases ◽  
Training Data ◽  
Model Accuracy ◽  
Healthcare Applications ◽  
Raw Data ◽  
Collaborative Training

Abstract Federated learning (FL) allows the collaborative training of AI models without needing to share raw data. This capability makes it especially interesting for healthcare applications where patient and data privacy is of utmost concern. However, recent works on the inversion of deep neural networks from model gradients raised concerns about the security of FL in preventing the leakage of training data. In this work, we show that these attacks presented in the literature are impractical in real FL use-cases and provide a new baseline attack that works for more realistic scenarios where the clients’ training involves updating the Batch Normalization (BN) statistics. Furthermore, we present new ways to measure and visualize potential data leakage in FL. Our work is a step towards establishing reproducible methods of measuring data leakage in FL and could help determine the optimal tradeoffs between privacy-preserving techniques, such as differential privacy, and model accuracy based on quantifiable metrics.

scholarly journals Towards Understanding the Risks of Gradient Inversion in Federated Learning

2021 ◽  
Author(s):  
Ali Hatamizadeh ◽  
Hongxu Yin ◽  
Pavlo Molchanov ◽  
Andriy Myronenko ◽  
Wenqi Li ◽  
...  
Keyword(s):  
Neural Networks ◽  
Data Privacy ◽  
Deep Neural Networks ◽  
Differential Privacy ◽  
Use Cases ◽  
Training Data ◽  
Model Accuracy ◽  
Healthcare Applications ◽  
Raw Data ◽  
Collaborative Training

Abstract Federated learning (FL) allows the collaborative training of AI models without needing to share raw data. This capability makes it especially interesting for healthcare applications where patient and data privacy is of utmost concern. However, recent works on the inversion of deep neural networks from model gradients raised concerns about the security of FL in preventing the leakage of training data. In this work, we show that these attacks presented in the literature are impractical in real FL use-cases and provide a new baseline attack that works for more realistic scenarios where the clients’ training involves updating the Batch Normalization (BN) statistics. Furthermore, we present new ways to measure and visualize potential data leakage in FL. Our work is a step towards establishing reproducible methods of measuring data leakage in FL and could help determine the optimal tradeoffs between privacy-preserving techniques, such as differential privacy, and model accuracy based on quantifiable metrics.

scholarly journals Scalable and Generalizable Social Bot Detection through Data Selection

2020 ◽  
Vol 34 (01) ◽  
pp. 1096-1103 ◽  
Author(s):  
Kai-Cheng Yang ◽  
Onur Varol ◽  
Pik-Mai Hui ◽  
Filippo Menczer
Keyword(s):  
Cross Validation ◽  
State Of The Art ◽  
Rapid Development ◽  
Model Performance ◽  
Training Data ◽  
Model Accuracy ◽  
Proposed Model ◽  
Information Manipulation ◽  
Bot Detection ◽  
Development State

Efficient and reliable social bot classification is crucial for detecting information manipulation on social media. Despite rapid development, state-of-the-art bot detection models still face generalization and scalability challenges, which greatly limit their applications. In this paper we propose a framework that uses minimal account metadata, enabling efficient analysis that scales up to handle the full stream of public tweets of Twitter in real time. To ensure model accuracy, we build a rich collection of labeled datasets for training and validation. We deploy a strict validation system so that model performance on unseen datasets is also optimized, in addition to traditional cross-validation. We find that strategically selecting a subset of training data yields better model accuracy and generalization than exhaustively training on all available data. Thanks to the simplicity of the proposed model, its logic can be interpreted to provide insights into social bot characteristics.

scholarly journals Federated Model Distillation with Noise-Free Differential Privacy

2021 ◽  
Author(s):  
Lichao Sun ◽  
Lingjuan Lyu
Keyword(s):  
Data Privacy ◽  
Differential Privacy ◽  
Random Noise ◽  
Model Performance ◽  
Training Data ◽  
Heterogeneous Model ◽  
The Public ◽  
Inference Attacks ◽  
Public Datasets ◽  
Privacy Budget

Conventional federated learning directly averages model weights, which is only possible for collaboration between models with homogeneous architectures. Sharing prediction instead of weight removes this obstacle and eliminates the risk of white-box inference attacks in conventional federated learning. However, the predictions from local models are sensitive and would leak training data privacy to the public. To address this issue, one naive approach is adding the differentially private random noise to the predictions, which however brings a substantial trade-off between privacy budget and model performance. In this paper, we propose a novel framework called FEDMD-NFDP, which applies a Noise-FreeDifferential Privacy (NFDP) mechanism into a federated model distillation framework. Our extensive experimental results on various datasets validate that FEDMD-NFDP can deliver not only comparable utility and communication efficiency but also provide a noise-free differential privacy guarantee. We also demonstrate the feasibility of our FEDMD-NFDP by considering both IID and Non-IID settings, heterogeneous model architectures, and unlabelled public datasets from a different distribution.

scholarly journals Quality control stress test for deep learning-based diagnostic model in digital pathology

2021 ◽  
Author(s):  
Birgid Schömig-Markiefka ◽  
Alexey Pryalukhin ◽  
Wolfgang Hulla ◽  
Andrey Bychkov ◽  
Junya Fukuoka ◽  
...  
Keyword(s):  
Deep Learning ◽  
Cancer Detection ◽  
Stress Testing ◽  
Computational Analysis ◽  
Stress Test ◽  
Digital Pathology ◽  
Model Performance ◽  
Diagnostic Model ◽  
Model Accuracy ◽  
Diagnostic Models

AbstractDigital pathology provides a possibility for computational analysis of histological slides and automatization of routine pathological tasks. Histological slides are very heterogeneous concerning staining, sections’ thickness, and artifacts arising during tissue processing, cutting, staining, and digitization. In this study, we digitally reproduce major types of artifacts. Using six datasets from four different institutions digitized by different scanner systems, we systematically explore artifacts’ influence on the accuracy of the pre-trained, validated, deep learning-based model for prostate cancer detection in histological slides. We provide evidence that any histological artifact dependent on severity can lead to a substantial loss in model performance. Strategies for the prevention of diagnostic model accuracy losses in the context of artifacts are warranted. Stress-testing of diagnostic models using synthetically generated artifacts might be an essential step during clinical validation of deep learning-based algorithms.

A Novel Cloud-Assisted Secure Deep Feature Classification Framework for Cancer Histopathology Images

2021 ◽  
Vol 21 (2) ◽  
pp. 1-22
Author(s):  
Abhinav Kumar ◽  
Sanjay Kumar Singh ◽  
K Lakshmanan ◽  
Sonal Saxena ◽  
Sameer Shrivastava
Keyword(s):  
Differential Privacy ◽  
Model Performance ◽  
Healthcare Services ◽  
Cloud Services ◽  
Efficient Computation ◽  
Sensitive Data ◽  
Classification Framework ◽  
Proposed Model ◽  
Diagnostic Framework ◽  
Histopathological Image Classification

The advancements in the Internet of Things (IoT) and cloud services have enabled the availability of smart e-healthcare services in a distant and distributed environment. However, this has also raised major privacy and efficiency concerns that need to be addressed. While sharing clinical data across the cloud that often consists of sensitive patient-related information, privacy is a major challenge. Adequate protection of patients’ privacy helps to increase public trust in medical research. Additionally, DL-based models are complex, and in a cloud-based approach, efficient data processing in such models is complicated. To address these challenges, we propose an efficient and secure cancer diagnostic framework for histopathological image classification by utilizing both differential privacy and secure multi-party computation. For efficient computation, instead of performing the whole operation on the cloud, we decouple the layers into two modules: one for feature extraction using the VGGNet module at the user side and the remaining layers for private prediction over the cloud. The efficacy of the framework is validated on two datasets composed of histopathological images of the canine mammary tumor and human breast cancer. The application of differential privacy preserving to the proposed model makes the model secure and capable of preserving the privacy of sensitive data from any adversary, without significantly compromising the model accuracy. Extensive experiments show that the proposed model efficiently achieves the trade-off between privacy and model performance.

scholarly journals Performance Evaluation of Deep CNN-Based Crack Detection and Localization Techniques for Concrete Structures

Sensors ◽  
2021 ◽  
Vol 21 (5) ◽  
pp. 1688
Author(s):  
Luqman Ali ◽  
Fady Alnajjar ◽  
Hamad Al Jassmi ◽  
Munkhjargal Gochoo ◽  
Wasif Khan ◽  
...  
Keyword(s):  
Neural Network ◽  
Convolutional Neural Network ◽  
Crack Detection ◽  
Concrete Structures ◽  
Model Performance ◽  
Training Data ◽  
Computational Time ◽  
Data Heterogeneity ◽  
Public Datasets ◽  
Detection And Localization

This paper proposes a customized convolutional neural network for crack detection in concrete structures. The proposed method is compared to four existing deep learning methods based on training data size, data heterogeneity, network complexity, and the number of epochs. The performance of the proposed convolutional neural network (CNN) model is evaluated and compared to pretrained networks, i.e., the VGG-16, VGG-19, ResNet-50, and Inception V3 models, on eight datasets of different sizes, created from two public datasets. For each model, the evaluation considered computational time, crack localization results, and classification measures, e.g., accuracy, precision, recall, and F1-score. Experimental results demonstrated that training data size and heterogeneity among data samples significantly affect model performance. All models demonstrated promising performance on a limited number of diverse training data; however, increasing the training data size and reducing diversity reduced generalization performance, and led to overfitting. The proposed customized CNN and VGG-16 models outperformed the other methods in terms of classification, localization, and computational time on a small amount of data, and the results indicate that these two models demonstrate superior crack detection and localization for concrete structures.

scholarly journals Modelling Exposure by Spraying Activities—Status and Future Needs

2021 ◽  
Vol 18 (15) ◽  
pp. 7737
Author(s):  
Stefan Hahn ◽  
Jessica Meyer ◽  
Michael Roitzsch ◽  
Christiaan Delmaar ◽  
Wolfgang Koch ◽  
...  
Keyword(s):  
Current Knowledge ◽  
Model Performance ◽  
Relevant Information ◽  
Systematic Evaluation ◽  
Model Accuracy ◽  
Efficient Manner ◽  
User Perspective ◽  
Model Predictions ◽  
Spray Applications

Spray applications enable a uniform distribution of substances on surfaces in a highly efficient manner, and thus can be found at workplaces as well as in consumer environments. A systematic literature review on modelling exposure by spraying activities has been conducted and status and further needs have been discussed with experts at a symposium. This review summarizes the current knowledge about models and their level of conservatism and accuracy. We found that extraction of relevant information on model performance for spraying from published studies and interpretation of model accuracy proved to be challenging, as the studies often accounted for only a small part of potential spray applications. To achieve a better quality of exposure estimates in the future, more systematic evaluation of models is beneficial, taking into account a representative variety of spray equipment and application patterns. Model predictions could be improved by more accurate consideration of variation in spray equipment. Inter-model harmonization with regard to spray input parameters and appropriate grouping of spray exposure situations is recommended. From a user perspective, a platform or database with information on different spraying equipment and techniques and agreed standard parameters for specific spraying scenarios from different regulations may be useful.

scholarly journals A Clonal Selection Optimization System for Multiparty Secure Computing

Complexity ◽  
2021 ◽  
Vol 2021 ◽  
pp. 1-14
Author(s):  
Minyu Shi ◽  
Yongting Zhang ◽  
Huanhuan Wang ◽  
Junfeng Hu ◽  
Xiang Wu
Keyword(s):  
Deep Learning ◽  
Large Scale ◽  
Differential Privacy ◽  
Clonal Selection ◽  
Smart Cities ◽  
Model Performance ◽  
Processing System ◽  
Optimization System ◽  
Learning Scheme ◽  
Training Efficiency

The innovation of the deep learning modeling scheme plays an important role in promoting the research of complex problems handled with artificial intelligence in smart cities and the development of the next generation of information technology. With the widespread use of smart interactive devices and systems, the exponential growth of data volume and the complex modeling requirements increase the difficulty of deep learning modeling, and the classical centralized deep learning modeling scheme has encountered bottlenecks in the improvement of model performance and the diversification of smart application scenarios. The parallel processing system in deep learning links the virtual information space with the physical world, although the distributed deep learning research has become a crucial concern with its unique advantages in training efficiency, and improving the availability of trained models and preventing privacy disclosure are still the main challenges faced by related research. To address these above issues in distributed deep learning, this research developed a clonal selective optimization system based on the federated learning framework for the model training process involving large-scale data. This system adopts the heuristic clonal selective strategy in local model optimization and optimizes the effect of federated training. First of all, this process enhances the adaptability and robustness of the federated learning scheme and improves the modeling performance and training efficiency. Furthermore, this research attempts to improve the privacy security defense capability of the federated learning scheme for big data through differential privacy preprocessing. The simulation results show that the proposed clonal selection optimization system based on federated learning has significant optimization ability on model basic performance, stability, and privacy.

Sign in / Sign up

Export Citation Format

Share Document