Application of Machine Learning for Oilfield Data Quality Improvement

Sensor platforms used in environmental monitoring applications are often subject to harsh environmental conditions while monitoring complex phenomena. Therefore, designing dependable monitoring systems is challenging given the external disturbances affecting sensor measurements. Even the apparently simple task of outlier detection in sensor data becomes a hard problem, amplified by the difficulty in distinguishing true data errors due to sensor faults from deviations due to natural phenomenon, which look like data errors. Existing solutions for runtime outlier detection typically assume that the physical processes can be accurately modeled, or that outliers consist in large deviations that are easily detected and filtered by appropriate thresholds. Other solutions assume that it is possible to deploy multiple sensors providing redundant data to support voting-based techniques. In this article, we propose a new methodology for dependable runtime detection of outliers in environmental monitoring systems, aiming to increase data quality by treating them. We propose the use of machine learning techniques to model each sensor behavior, exploiting the existence of correlated data provided by other related sensors. Using these models, along with knowledge of processed past measurements, it is possible to obtain accurate estimations of the observed environment parameters and build failure detectors that use these estimations. When a failure is detected, these estimations also allow one to correct the erroneous measurements and hence improve the overall data quality. Our methodology not only allows one to distinguish truly abnormal measurements from deviations due to complex natural phenomena, but also allows the quantification of each measurement quality, which is relevant from a dependability perspective. We apply the methodology to real datasets from a complex aquatic monitoring system, measuring temperature and salinity parameters, through which we illustrate the process for building the machine learning prediction models using a technique based on Artificial Neural Networks, denoted ANNODE ( ANN Outlier Detection ). From this application, we also observe the effectiveness of our ANNODE approach for accurate outlier detection in harsh environments. Then we validate these positive results by comparing ANNODE with state-of-the-art solutions for outlier detection. The results show that ANNODE improves existing solutions regarding accuracy of outlier detection.

Download Full-text

Perspective of anomaly detection in big data for data quality improvement

Materials Today Proceedings ◽

10.1016/j.matpr.2021.05.597 ◽

2021 ◽

Author(s):

Vinaya Keskar ◽

Jyoti Yadav ◽

Ajay Kumar

Keyword(s):

Big Data ◽

Quality Improvement ◽

Anomaly Detection ◽

Data Quality

Download Full-text

Data Quality Measures and Efficient Evaluation Algorithms for Large-Scale High-Dimensional Data

Applied Sciences ◽

10.3390/app11020472 ◽

2021 ◽

Vol 11 (2) ◽

pp. 472

Author(s):

Hyeongmin Cho ◽

Sangkyun Lee

Keyword(s):

Machine Learning ◽

Data Quality ◽

Large Scale ◽

High Dimensional Data ◽

Quality Measures ◽

Training Data ◽

Measure Data ◽

High Dimensional ◽

Small Scale ◽

Class Separability

Machine learning has been proven to be effective in various application areas, such as object and speech recognition on mobile systems. Since a critical key to machine learning success is the availability of large training data, many datasets are being disclosed and published online. From a data consumer or manager point of view, measuring data quality is an important first step in the learning process. We need to determine which datasets to use, update, and maintain. However, not many practical ways to measure data quality are available today, especially when it comes to large-scale high-dimensional data, such as images and videos. This paper proposes two data quality measures that can compute class separability and in-class variability, the two important aspects of data quality, for a given dataset. Classical data quality measures tend to focus only on class separability; however, we suggest that in-class variability is another important data quality factor. We provide efficient algorithms to compute our quality measures based on random projections and bootstrapping with statistical benefits on large-scale high-dimensional data. In experiments, we show that our measures are compatible with classical measures on small-scale data and can be computed much more efficiently on large-scale high-dimensional datasets.

Download Full-text

A Machine Learning Approach for Data Quality Control of Earth Observation Data Management System

IGARSS 2020 - 2020 IEEE International Geoscience and Remote Sensing Symposium ◽

10.1109/igarss39084.2020.9323615 ◽

2020 ◽

Author(s):

Weiguo Han ◽

Matthew Jochum

Keyword(s):

Machine Learning ◽

Quality Control ◽

Data Quality ◽

Earth Observation ◽

Data Management System ◽

Learning Approach ◽

Observation Data ◽

Data Quality Control ◽

Machine Learning Approach ◽

Earth Observation Data

Download Full-text

Improving Power Grid Monitoring Data Quality: An Efficient Machine Learning Framework for Missing Data Prediction

2015 IEEE 17th International Conference on High Performance Computing and Communications, 2015 IEEE 7th International Symposium on Cyberspace Safety and Security, and 2015 IEEE 12th International Conference on Embedded Software and Systems ◽

10.1109/hpcc-css-icess.2015.16 ◽

2015 ◽

Cited By ~ 10

Author(s):

Weiwei Shi ◽

Yongxin Zhu ◽

Jinkui Zhang ◽

Xiang Tao ◽

Gehao Sheng ◽

...

Keyword(s):

Machine Learning ◽

Missing Data ◽

Data Quality ◽

Power Grid ◽

Monitoring Data ◽

Learning Framework ◽

Data Prediction ◽

Grid Monitoring ◽

Efficient Machine ◽

Missing Data Prediction

Download Full-text

Application of Machine Learning for Lithology-on-Bit Prediction using Drilling Data in Real-Time

10.2118/206622-ms ◽

2021 ◽

Author(s):

Temirlan Zhekenov ◽

Artem Nechaev ◽

Kamilla Chettykbayeva ◽

Alexey Zinovyev ◽

German Sardarov ◽

...

Keyword(s):

Machine Learning ◽

Data Quality ◽

Real Time ◽

Hybrid Modeling ◽

Data Set ◽

Drilling Parameters ◽

Drilling Data ◽

Stable Solutions ◽

Mud Logging

SUMMARY Researchers base their analysis on basic drilling parameters obtained during mud logging and demonstrate impressive results. However, due to limitations imposed by data quality often present during drilling, those solutions often tend to lose their stability and high levels of predictivity. In this work, the concept of hybrid modeling was introduced which allows to integrate the analytical correlations with algorithms of machine learning for obtaining stable solutions consistent from one data set to another.

Download Full-text

Missing Observations and Data Quality Improvement

Handling Missing Data in Ranked Set Sampling - SpringerBriefs in Statistics ◽

10.1007/978-3-642-39899-5_1 ◽

2013 ◽

pp. 1-6

Author(s):

Carlos N. Bouza-Herrera

Keyword(s):

Quality Improvement ◽

Data Quality ◽

Missing Observations

Download Full-text

Drilling Data Quality Management: Case Study With a Laboratory Scale Drilling Rig

Volume 8: Polar and Arctic Sciences and Technology; Petroleum Technology ◽

10.1115/omae2018-77510 ◽

2018 ◽

Cited By ~ 1

Author(s):

Suranga C. H. Geekiyanage ◽

Dan Sui ◽

Bernt S. Aadnoy

Keyword(s):

Decision Making ◽

Quality Improvement ◽

Control System ◽

Data Analysis ◽

Quality Management ◽

Data Quality ◽

Incomplete Data ◽

Laboratory Scale ◽

Data Quality Management ◽

Drilling Rig

Drilling industry operations heavily depend on digital information. Data analysis is a process of acquiring, transforming, interpreting, modelling, displaying and storing data with an aim of extracting useful information, so that the decision-making, actions executing, events detecting and incident managing of a system can be handled in an efficient and certain manner. This paper aims to provide an approach to understand, cleanse, improve and interpret the post-well or realtime data to preserve or enhance data features, like accuracy, consistency, reliability and validity. Data quality management is a process with three major phases. Phase I is an evaluation of pre-data quality to identify data issues such as missing or incomplete data, non-standard or invalid data and redundant data etc. Phase II is an implementation of different data quality managing practices such as filtering, data assimilation, and data reconciliation to improve data accuracy and discover useful information. The third and final phase is a post-data quality evaluation, which is conducted to assure data quality and enhance the system performance. In this study, a laboratory-scale drilling rig with a control system capable of drilling is utilized for data acquisition and quality improvement. Safe and efficient performance of such control system heavily relies on quality of the data obtained while drilling and its sufficient availability. Pump pressure, top-drive rotational speed, weight on bit, drill string torque and bit depth are available measurements. The data analysis is challenged by issues such as corruption of data due to noises, time delays, missing or incomplete data and external disturbances. In order to solve such issues, different data quality improvement practices are applied for the testing. These techniques help the intelligent system to achieve better decision-making and quicker fault detection. The study from the laboratory-scale drilling rig clearly demonstrates the need for a proper data quality management process and clear understanding of signal processing methods to carry out an intelligent digitalization in oil and gas industry.

Download Full-text