Distributed and automated machine learning in big data stream analytics

2019 ◽  
Author(s):  
Oraib H. Alsahlee ◽  
Abdallah Al-Zu'bi ◽  
Ibrahim I. Alamro ◽  
Jaber Alwidian
Keyword(s):  
Machine Learning ◽  
Big Data ◽  
Data Stream ◽  
Automated Machine Learning ◽  
Stream Analytics

scholarly journals A Survey on Data Analysis on Large-Scale Wireless Networks: Online Stream Processing, Trends, and Challenges

2020 ◽  
Author(s):  
Dianne Scherly Varela de Medeiros ◽  
Helio do Nascimento Cunha Neto ◽  
Martin Andreoni Lopez ◽  
Luiz Claudio Schara Magalhães ◽  
Natalia Castro Fernandes ◽  
...  
Keyword(s):  
Machine Learning ◽  
Wireless Networks ◽  
Big Data ◽  
Wireless Network ◽  
Data Stream ◽  
Large Scale ◽  
Stream Processing ◽  
Knowledge Extraction ◽  
Machine Learning Algorithms ◽  
Data Stream Processing

Abstract In this paper we focus on knowledge extraction from large-scale wireless networks through stream processing. We present the primary methods for sampling, data collection, and monitoring of wireless networks and we characterize knowledge extraction as a machine learning problem on big data stream processing. We show the main trends in big data stream processing frameworks. Additionally, we explore the data preprocessing, feature engineering, and the machine learning algorithms applied to the scenario of wireless network analytics. We address challenges and present research projects in wireless network monitoring and stream processing. Finally, future perspectives, such as deep learning and reinforcement learning in stream processing, are anticipated.

scholarly journals A Survey on Data Analysis on Large-Scale Wireless Networks: Online Stream Processing, Trends, and Challenges

2020 ◽  
Author(s):  
Dianne Scherly Varela de Medeiros ◽  
Helio do Nascimento Cunha Neto ◽  
Martin Andreoni Lopez ◽  
Luiz Claudio Schara Magalhães ◽  
Natalia Castro Fernandes ◽  
...  
Keyword(s):  
Machine Learning ◽  
Wireless Networks ◽  
Big Data ◽  
Wireless Network ◽  
Data Stream ◽  
Large Scale ◽  
Stream Processing ◽  
Knowledge Extraction ◽  
Machine Learning Algorithms ◽  
Data Stream Processing

Abstract In this paper we focus on knowledge extraction from large-scale wireless networks through stream processing. We present the primary methods for sampling, data collection, and monitoring of wireless networks and we characterize knowledge extraction as a machine learning problem on big data stream processing. We show the main trends in big data stream processing frameworks. Additionally, we explore the data preprocessing, feature engineering, and the machine learning algorithms applied to the scenario of wireless network analytics. We address challenges and present research projects in wireless network monitoring and stream processing. Finally, future perspectives, such as deep learning and reinforcement learning in stream processing, are anticipated.

scholarly journals Embedding covariate adjustments in tree-based automated machine learning for biomedical big data analyses

2020 ◽  
Vol 21 (1) ◽  
Author(s):  
Elisabetta Manduchi ◽  
Weixuan Fu ◽  
Joseph D. Romano ◽  
Stefano Ruberto ◽  
Jason H. Moore
Keyword(s):  
Gene Expression ◽  
Machine Learning ◽  
Big Data ◽  
Gene Expression Data ◽  
Data Sets ◽  
Biomedical Data ◽  
Training Procedure ◽  
Expression Data ◽  
Differential Gene ◽  
Automated Machine Learning

Abstract Background A typical task in bioinformatics consists of identifying which features are associated with a target outcome of interest and building a predictive model. Automated machine learning (AutoML) systems such as the Tree-based Pipeline Optimization Tool (TPOT) constitute an appealing approach to this end. However, in biomedical data, there are often baseline characteristics of the subjects in a study or batch effects that need to be adjusted for in order to better isolate the effects of the features of interest on the target. Thus, the ability to perform covariate adjustments becomes particularly important for applications of AutoML to biomedical big data analysis. Results We developed an approach to adjust for covariates affecting features and/or target in TPOT. Our approach is based on regressing out the covariates in a manner that avoids ‘leakage’ during the cross-validation training procedure. We describe applications of this approach to toxicogenomics and schizophrenia gene expression data sets. The TPOT extensions discussed in this work are available at https://github.com/EpistasisLab/tpot/tree/v0.11.1-resAdj. Conclusions In this work, we address an important need in the context of AutoML, which is particularly crucial for applications to bioinformatics and medical informatics, namely covariate adjustments. To this end we present a substantial extension of TPOT, a genetic programming based AutoML approach. We show the utility of this extension by applications to large toxicogenomics and differential gene expression data. The method is generally applicable in many other scenarios from the biomedical field.

Sign in / Sign up

Export Citation Format

Share Document