An Innovative Lambda-Architecture-Based Data Warehouse Maintenance Framework for Effective and Efficient Near-Real-Time OLAP over Big Data

2018 ◽  
pp. 149-165 ◽  
Author(s):  
Alfredo Cuzzocrea ◽  
Rim Moussa ◽  
Gianni Vercelli
Keyword(s):  
Big Data ◽  
Real Time ◽  
Data Warehouse ◽  
Lambda Architecture

scholarly journals Multi-Agent Big-Data Lambda Architecture Model for E-Commerce Analytics

Data ◽  
2018 ◽  
Vol 3 (4) ◽  
pp. 58 ◽  
Author(s):  
Gautam Pal ◽  
Gangmin Li ◽  
Katie Atkinson
Keyword(s):  
Big Data ◽  
Real Time ◽  
High Velocity ◽  
Turnaround Time ◽  
Low Latency ◽  
Multi Agent Systems ◽  
Architecture Model ◽  
Lambda Architecture ◽  
Agent Interaction ◽  
Multi Agent

We study big-data hybrid-data-processing lambda architecture, which consolidates low-latency real-time frameworks with high-throughput Hadoop-batch frameworks over a massively distributed setup. In particular, real-time and batch-processing engines act as autonomous multi-agent systems in collaboration. We propose a Multi-Agent Lambda Architecture (MALA) for e-commerce data analytics. We address the high-latency problem of Hadoop MapReduce jobs by simultaneous processing at the speed layer to the requests which require a quick turnaround time. At the same time, the batch layer in parallel provides comprehensive coverage of data by intelligent blending of stream and historical data through the weighted voting method. The cold-start problem of streaming services is addressed through the initial offset from historical batch data. Challenges of high-velocity data ingestion is resolved with distributed message queues. A proposed multi-agent decision-maker component is placed at the MALA stack as the gateway of the data pipeline. We prove efficiency of our batch model by implementing an array of features for an e-commerce site. The novelty of the model and its key significance is a scheme for multi-agent interaction between batch and real-time agents to produce deeper insights at low latency and at significantly lower costs. Hence, the proposed system is highly appealing for applications involving big data and caters to high-velocity streaming ingestion and a massive data pool.

scholarly journals A Frequency Pattern Mining Model Based on Deep Neural Network for Real-Time Classification of Heart Conditions

Healthcare ◽  
2020 ◽  
Vol 8 (3) ◽  
pp. 234 ◽  
Author(s):  
Hyun Yoo ◽  
Soyoung Han ◽  
Kyungyong Chung
Keyword(s):  
Neural Network ◽  
Big Data ◽  
Fourier Transform ◽  
Fast Fourier Transform ◽  
Real Time ◽  
Normal Control ◽  
Input Data ◽  
Deep Neural Network ◽  
Pattern Mining ◽  
F Measure

Recently, a massive amount of big data of bioinformation is collected by sensor-based IoT devices. The collected data are also classified into different types of health big data in various techniques. A personalized analysis technique is a basis for judging the risk factors of personal cardiovascular disorders in real-time. The objective of this paper is to provide the model for the personalized heart condition classification in combination with the fast and effective preprocessing technique and deep neural network in order to process the real-time accumulated biosensor input data. The model can be useful to learn input data and develop an approximation function, and it can help users recognize risk situations. For the analysis of the pulse frequency, a fast Fourier transform is applied in preprocessing work. With the use of the frequency-by-frequency ratio data of the extracted power spectrum, data reduction is performed. To analyze the meanings of preprocessed data, a neural network algorithm is applied. In particular, a deep neural network is used to analyze and evaluate linear data. A deep neural network can make multiple layers and can establish an operation model of nodes with the use of gradient descent. The completed model was trained by classifying the ECG signals collected in advance into normal, control, and noise groups. Thereafter, the ECG signal input in real time through the trained deep neural network system was classified into normal, control, and noise. To evaluate the performance of the proposed model, this study utilized a ratio of data operation cost reduction and F-measure. As a result, with the use of fast Fourier transform and cumulative frequency percentage, the size of ECG reduced to 1:32. According to the analysis on the F-measure of the deep neural network, the model had 83.83% accuracy. Given the results, the modified deep neural network technique can reduce the size of big data in terms of computing work, and it is an effective system to reduce operation time.

scholarly journals A Real Time Processing system for big data in astronomy: Applications to HERA

2021 ◽  
pp. 100489
Author(s):  
Paul La Plante ◽  
P.K.G. Williams ◽  
M. Kolopanis ◽  
J.S. Dillon ◽  
A.P. Beardsley ◽  
...  
Keyword(s):  
Big Data ◽  
Real Time ◽  
Processing System ◽  
Real Time Processing ◽  
Time Processing

scholarly journals Anomaly Identification during Polymerase Chain Reaction for Detecting SARS-CoV-2 Using Artificial Intelligence Trained from Simulated Data

Molecules ◽  
2020 ◽  
Vol 26 (1) ◽  
pp. 20
Author(s):  
Reynaldo Villarreal-González ◽  
Antonio J. Acosta-Hoyos ◽  
Jaime A. Garzon-Ochoa ◽  
Nataly J. Galán-Freyle ◽  
Paola Amar-Sepúlveda ◽  
...  
Keyword(s):  
Artificial Intelligence ◽  
Big Data ◽  
Real Time ◽  
Binary Classification ◽  
Simulated Data ◽  
Classification Model ◽  
Rt Pcr ◽  
Simon Bolivar ◽  
Polymerase Chain ◽  
Available Information

Real-time reverse transcription (RT) PCR is the gold standard for detecting Severe Acute Respiratory Syndrome Coronavirus 2 (SARS-CoV-2), owing to its sensitivity and specificity, thereby meeting the demand for the rising number of cases. The scarcity of trained molecular biologists for analyzing PCR results makes data verification a challenge. Artificial intelligence (AI) was designed to ease verification, by detecting atypical profiles in PCR curves caused by contamination or artifacts. Four classes of simulated real-time RT-PCR curves were generated, namely, positive, early, no, and abnormal amplifications. Machine learning (ML) models were generated and tested using small amounts of data from each class. The best model was used for classifying the big data obtained by the Virology Laboratory of Simon Bolivar University from real-time RT-PCR curves for SARS-CoV-2, and the model was retrained and implemented in a software that correlated patient data with test and AI diagnoses. The best strategy for AI included a binary classification model, which was generated from simulated data, where data analyzed by the first model were classified as either positive or negative and abnormal. To differentiate between negative and abnormal, the data were reevaluated using the second model. In the first model, the data required preanalysis through a combination of prepossessing. The early amplification class was eliminated from the models because the numbers of cases in big data was negligible. ML models can be created from simulated data using minimum available information. During analysis, changes or variations can be incorporated by generating simulated data, avoiding the incorporation of large amounts of experimental data encompassing all possible changes. For diagnosing SARS-CoV-2, this type of AI is critical for optimizing PCR tests because it enables rapid diagnosis and reduces false positives. Our method can also be used for other types of molecular analyses.

Decision Based Model for Real-Time IoT Analysis Using Big Data and Machine Learning

2021 ◽  
Author(s):  
Hina Jamil ◽  
Tariq Umer ◽  
Celal Ceken ◽  
Fadi Al-Turjman
Keyword(s):  
Machine Learning ◽  
Big Data ◽  
Real Time
Sign in / Sign up

Export Citation Format

Share Document