An online anomaly detection method for stream data using isolation principle and statistic histogram

2015 ◽  
Vol 06 (02) ◽  
pp. 1550017 ◽  
Author(s):  
Zhiguo Ding ◽  
Minrui Fei ◽  
Dajun Du
Keyword(s):  
Machine Learning ◽  
Anomaly Detection ◽  
Ensemble Learning ◽  
Detection Method ◽  
Dynamic Evolution ◽  
Stream Data ◽  
Massive Datasets ◽  
Online Anomaly Detection

Online anomaly detection for stream data has been explored recently, where the detector is supposed to be able to perform an accurate and timely judgment for the upcoming observation. However, due to the inherent complex characteristics of stream data, such as quick generation, tremendous volume and dynamic evolution distribution, how to develop an effective online anomaly detection method is a challenge. The main objective of this paper is to propose an adaptive online anomaly detection method for stream data. This is achieved by combining isolation principle with online ensemble learning, which is then optimized by statistic histogram. Three main algorithms are developed, i.e., online detector building algorithm, anomaly detecting algorithm and adaptive detector updating algorithm. To evaluate our proposed method, four massive datasets from the UCI machine learning repository recorded from real events were adopted. Extensive simulations based on these datasets show that our method is effective and robust against different scenarios.

scholarly journals Semi-Supervised Machine Learning Aided Anomaly Detection Method in Cellular Networks

2020 ◽  
Vol 69 (8) ◽  
pp. 8459-8467 ◽  
Author(s):  
Yutao Lu ◽  
Juan Wang ◽  
Miao Liu ◽  
Kaixuan Zhang ◽  
Guan Gui ◽  
...  
Keyword(s):  
Machine Learning ◽  
Anomaly Detection ◽  
Cellular Networks ◽  
Detection Method ◽  
Supervised Machine Learning

scholarly journals Danger-Pose Detection System Using Commodity Wi-Fi for Bathroom Monitoring

Sensors ◽  
2019 ◽  
Vol 19 (4) ◽  
pp. 884 ◽  
Author(s):  
Zizheng Zhang ◽  
Shigemi Ishida ◽  
Shigeaki Tagashira ◽  
Akira Fukuda
Keyword(s):  
Machine Learning ◽  
Anomaly Detection ◽  
Detection Method ◽  
Detection System ◽  
Wearable Sensors ◽  
Low Frequency ◽  
Training Data ◽  
Support Vector ◽  
Dynamic Features ◽  
Pose Detection

A bathroom has higher probability of accidents than other rooms due to a slippery floor and temperature change. Because of high privacy and humidity, we face difficulties in monitoring inside a bathroom using traditional healthcare methods based on cameras and wearable sensors. In this paper, we present a danger-pose detection system using commodity Wi-Fi devices, which can be applied to bathroom monitoring, preserving privacy. A machine learning-based detection method usually requires data collected in target situations, which is difficult in detection-of-danger situations. We therefore employ a machine learning-based anomaly-detection method that requires a small amount of data in anomaly conditions, minimizing the required training data collected in dangerous conditions. We first derive the amplitude and phase shift from Wi-Fi channel state information (CSI) to extract low-frequency components that are related to human activities. We then separately extract static and dynamic features from the CSI changes in time. Finally, the static and dynamic features are fed into a one-class support vector machine (SVM), which is used as an anomaly-detection method, to classify whether a user is not in bathtub, bathing safely, or in dangerous conditions. We conducted experimental evaluations and demonstrated that our danger-pose detection system achieved a high detection performance in a non-line-of-sight (NLOS) scenario.

scholarly journals A Novel Distributed Online Anomaly Detection Method in Resource-Constrained Wireless Sensor Networks

2015 ◽  
Vol 2015 ◽  
pp. 1-12
Author(s):  
Zhiguo Ding ◽  
Haikuan Wang ◽  
Minrui Fei ◽  
Dajun Du
Keyword(s):  
Anomaly Detection ◽  
Detection Method ◽  
Cluster Head ◽  
Sensor Nodes ◽  
Wireless Sensor ◽  
Resource Constrained ◽  
State Matrix ◽  
Head Node ◽  
Global Anomaly ◽  
Online Anomaly Detection

In this paper, a novel distributed online anomaly detection method in resource-constrained WSNs was proposed. Firstly, the spatiotemporal correlation existing in the sensed data was exploited and a series of single anomaly detectors were built in each distributed deployment sensor node based on ensemble learning theory. Secondly, these trained detectors were broadcasted to the member sensor nodes in the cluster, combining with its trained detector, and the initial ensemble detector was built. Thirdly, considering resources-constrained WSNs, ensemble pruning based on biogeographical based optimization (BBO) was employed in the cluster head node to obtain an optimized subset of ensemble members. Further, the pruned ensemble detector coded by the state matrix was broadcasted to each member sensor nodes for the distributed online global anomaly detection. Finally, the experiments operated on a real WSN dataset demonstrated the effectiveness of the proposed method.

VADAF: Visualization for Abnormal Client Detection and Analysis in Federated Learning

2021 ◽  
Vol 11 (3-4) ◽  
pp. 1-23
Author(s):  
Linhao Meng ◽  
Yating Wei ◽  
Rusheng Pan ◽  
Shuyue Zhou ◽  
Jianwei Zhang ◽  
...  
Keyword(s):  
Machine Learning ◽  
Anomaly Detection ◽  
Visual Analytics ◽  
Detection Method ◽  
Training Process ◽  
Local Data ◽  
Privacy And Security ◽  
Potential Client ◽  
Distributed Machine Learning ◽  
Large Corpus

Federated Learning (FL) provides a powerful solution to distributed machine learning on a large corpus of decentralized data. It ensures privacy and security by performing computation on devices (which we refer to as clients) based on local data to improve the shared global model. However, the inaccessibility of the data and the invisibility of the computation make it challenging to interpret and analyze the training process, especially to distinguish potential client anomalies. Identifying these anomalies can help experts diagnose and improve FL models. For this reason, we propose a visual analytics system, VADAF, to depict the training dynamics and facilitate analyzing potential client anomalies. Specifically, we design a visualization scheme that supports massive training dynamics in the FL environment. Moreover, we introduce an anomaly detection method to detect potential client anomalies, which are further analyzed based on both the client model’s visual and objective estimation. Three case studies have demonstrated the effectiveness of our system in understanding the FL training process and supporting abnormal client detection and analysis.

Streaming data anomaly detection method based on hyper-grid structure and online ensemble learning

2016 ◽  
Vol 21 (20) ◽  
pp. 5905-5917 ◽  
Author(s):  
Zhiguo Ding ◽  
Minrui Fei ◽  
Dajun Du ◽  
Fan Yang
Keyword(s):  
Anomaly Detection ◽  
Ensemble Learning ◽  
Detection Method ◽  
Streaming Data ◽  
Grid Structure

scholarly journals Combining Machine Learning Models Using combo Library

2020 ◽  
Vol 34 (09) ◽  
pp. 13648-13649
Author(s):  
Yue Zhao ◽  
Xuejian Wang ◽  
Cheng Cheng ◽  
Xueying Ding
Keyword(s):  
Machine Learning ◽  
Anomaly Detection ◽  
Ensemble Learning ◽  
Academic Research ◽  
Learning Models ◽  
Model Combination ◽  
Code Coverage ◽  
Industry Applications ◽  
Python Package ◽  
Machine Learning Models

Model combination, often regarded as a key sub-field of ensemble learning, has been widely used in both academic research and industry applications. To facilitate this process, we propose and implement an easy-to-use Python toolkit, combo, to aggregate models and scores under various scenarios, including classification, clustering, and anomaly detection. In a nutshell, combo provides a unified and consistent way to combine both raw and pretrained models from popular machine learning libraries, e.g., scikit-learn, XGBoost, and LightGBM. With accessibility and robustness in mind, combo is designed with detailed documentation, interactive examples, continuous integration, code coverage, and maintainability check; it can be installed easily through Python Package Index (PyPI) or {https://github.com/yzhao062/combo}.

Sign in / Sign up

Export Citation Format

Share Document