scholarly journals Kurz erklärt: Measuring Data Changes in Data Engineering and their Impact on Explainability and Algorithm Fairness

2021 ◽  
Author(s):  
Meike Klettke ◽  
Adrian Lutsch ◽  
Uta Störl
Keyword(s):  
Machine Learning ◽  
Data Quality ◽  
Data Science ◽  
Data Preprocessing ◽  
Learning Processes ◽  
Improve Data Quality ◽  
Data Engineering ◽  
Research Questions

AbstractData engineering is an integral part of any data science and ML process. It consists of several subtasks that are performed to improve data quality and to transform data into a target format suitable for analysis. The quality and correctness of the data engineering steps is therefore important to ensure the quality of the overall process.In machine learning processes requirements such as fairness and explainability are essential. The answers to these must also be provided by the data engineering subtasks. In this article, we will show how these can be achieved by logging, monitoring and controlling the data changes in order to evaluate their correctness. However, since data preprocessing algorithms are part of any machine learning pipeline, they must obviously also guarantee that they do not produce data biases.In this article we will briefly introduce three classes of methods for measuring data changes in data engineering and present which research questions still remain unanswered in this area.

Data science in economics: comprehensive review of advanced machine learning and deep learning methods

2020 ◽  
Author(s):  
Saeed Nosratabadi ◽  
Amir Mosavi ◽  
Puhong Duan ◽  
Pedram Ghamisi ◽  
Ferdinand Filip ◽  
...  
Keyword(s):  
Machine Learning ◽  
Deep Learning ◽  
Data Science ◽  
State Of The Art ◽  
Science Methods ◽  
Learning Models ◽  
Diverse Range ◽  
Hybrid Machine ◽  
Economics Research

This paper provides a state-of-the-art investigation of advances in data science in emerging economic applications. The analysis was performed on novel data science methods in four individual classes of deep learning models, hybrid deep learning models, hybrid machine learning, and ensemble models. Application domains include a wide and diverse range of economics research from the stock market, marketing, and e-commerce to corporate banking and cryptocurrency. Prisma method, a systematic literature review methodology, was used to ensure the quality of the survey. The findings reveal that the trends follow the advancement of hybrid models, which, based on the accuracy metric, outperform other learning algorithms. It is further expected that the trends will converge toward the advancements of sophisticated hybrid deep learning models.

scholarly journals AI Enabled Data Quality Monitoring with Hydra

2021 ◽  
Vol 251 ◽  
pp. 04010
Author(s):  
Thomas Britton ◽  
David Lawrence ◽  
Kishansingh Rajput
Keyword(s):  
Artificial Intelligence ◽  
Data Quality ◽  
Data Science ◽  
Quality Monitoring ◽  
Alarm System ◽  
Future Outlook ◽  
Skill Sets ◽  
Early Results ◽  
Level Monitoring

Data quality monitoring is critical to all experiments impacting the quality of any physics results. Traditionally, this is done through an alarm system, which detects low level faults, leaving higher level monitoring to human crews. Artificial Intelligence is beginning to find its way into scientific applications, but comes with difficulties, relying on the acquisition of new skill sets, either through education or acquisition, in data science. This paper will discuss the development and deployment of the Hydra monitoring system in production at Gluex. It will show how “off-the-shelf” technologies can be rapidly developed, as well as discuss what sociological hurdles must be overcome to successfully deploy such a system. Early results from production running of Hydra will also be shared as well as a future outlook for development of Hydra.

Integrating Experimental and Analytic Approaches to Improve Data Quality in Genome-wide RNAi Screens

2008 ◽  
Vol 13 (5) ◽  
pp. 378-389 ◽  
Author(s):  
Xiaohua Douglas Zhang ◽  
Amy S. Espeseth ◽  
Eric N. Johnson ◽  
Jayne Chin ◽  
Adam Gates ◽  
...  
Keyword(s):  
Data Quality ◽  
High Throughput Screening ◽  
Daily Practice ◽  
Systematic Errors ◽  
Quality Of Data ◽  
Improve Data Quality ◽  
Research Journal ◽  
Genome Wide ◽  
Assess Data Quality

RNA interference (RNAi) not only plays an important role in drug discovery but can also be developed directly into drugs. RNAi high-throughput screening (HTS) biotechnology allows us to conduct genome-wide RNAi research. A central challenge in genome-wide RNAi research is to integrate both experimental and computational approaches to obtain high quality RNAi HTS assays. Based on our daily practice in RNAi HTS experiments, we propose the implementation of 3 experimental and analytic processes to improve the quality of data from RNAi HTS biotechnology: (1) select effective biological controls; (2) adopt appropriate plate designs to display and/or adjust for systematic errors of measurement; and (3) use effective analytic metrics to assess data quality. The applications in 5 real RNAi HTS experiments demonstrate the effectiveness of integrating these processes to improve data quality. Due to the effectiveness in improving data quality in RNAi HTS experiments, the methods and guidelines contained in the 3 experimental and analytic processes are likely to have broad utility in genome-wide RNAi research. ( Journal of Biomolecular Screening 2008:378-389)

Improving Real-Time Drilling Data Quality Using Artificial Intelligence and Machine Learning Techniques

2021 ◽  
Author(s):  
S. H. Al Gharbi ◽  
A. A. Al-Majed ◽  
A. Abdulraheem ◽  
S. Patil ◽  
S. M. Elkatatny
Keyword(s):  
Artificial Intelligence ◽  
Machine Learning ◽  
Data Quality ◽  
Real Time ◽  
Input Data ◽  
Support Vector ◽  
The Real ◽  
Drilling Data ◽  
Drilling Operations

Abstract Due to high demand for energy, oil and gas companies started to drill wells in remote areas and unconventional environments. This raised the complexity of drilling operations, which were already challenging and complex. To adapt, drilling companies expanded their use of the real-time operation center (RTOC) concept, in which real-time drilling data are transmitted from remote sites to companies’ headquarters. In RTOC, groups of subject matter experts monitor the drilling live and provide real-time advice to improve operations. With the increase of drilling operations, processing the volume of generated data is beyond a human's capability, limiting the RTOC impact on certain components of drilling operations. To overcome this limitation, artificial intelligence and machine learning (AI/ML) technologies were introduced to monitor and analyze the real-time drilling data, discover hidden patterns, and provide fast decision-support responses. AI/ML technologies are data-driven technologies, and their quality relies on the quality of the input data: if the quality of the input data is good, the generated output will be good; if not, the generated output will be bad. Unfortunately, due to the harsh environments of drilling sites and the transmission setups, not all of the drilling data is good, which negatively affects the AI/ML results. The objective of this paper is to utilize AI/ML technologies to improve the quality of real-time drilling data. The paper fed a large real-time drilling dataset, consisting of over 150,000 raw data points, into Artificial Neural Network (ANN), Support Vector Machine (SVM) and Decision Tree (DT) models. The models were trained on the valid and not-valid datapoints. The confusion matrix was used to evaluate the different AI/ML models including different internal architectures. Despite the slowness of ANN, it achieved the best result with an accuracy of 78%, compared to 73% and 41% for DT and SVM, respectively. The paper concludes by presenting a process for using AI technology to improve real-time drilling data quality. To the author's knowledge based on literature in the public domain, this paper is one of the first to compare the use of multiple AI/ML techniques for quality improvement of real-time drilling data. The paper provides a guide for improving the quality of real-time drilling data.

Automated Machine Learning for Business

2021 ◽  
Author(s):  
Kai R. Larsen ◽  
Daniel S. Becker
Keyword(s):  
Machine Learning ◽  
Data Science ◽  
Graduate Program ◽  
School Curriculum ◽  
High School Curriculum ◽  
Stakeholder Groups ◽  
New Development ◽  
Automated Machine Learning ◽  
Undergraduate Programs

In Automated Machine Learning for Business, we teach the machine learning process using a new development in data science: automated machine learning. AutoML, when implemented properly, makes machine learning accessible to most people because it removes the need for years of experience in the most arcane aspects of data science, such as the math, statistics, and computer science skills required to become a top contender in traditional machine learning. Anyone trained in the use of AutoML can use it to test their ideas and support the quality of those ideas during presentations to management and stakeholder groups. Because the requisite investment is one semester-long undergraduate course rather than a year in a graduate program, these tools will likely become a core component of undergraduate programs, and over time, even the high school curriculum.

Data science in economics: comprehensive review of advanced machine learning and deep learning methods

2020 ◽  
Author(s):  
Saeed Nosratabadi ◽  
Amir Mosavi ◽  
Puhong Duan ◽  
Pedram Ghamisi ◽  
Filip Ferdinand ◽  
...  
Keyword(s):  
Machine Learning ◽  
Deep Learning ◽  
Prediction Accuracy ◽  
Data Science ◽  
State Of The Art ◽  
Hybrid Models ◽  
The Other ◽  
Learning Models ◽  
Comprehensive Review

Abstract This paper provides the state of the art of data science in economics. Through a novel taxonomy of applications and methods advances in data science are investigated. The data science advances are investigated in three individual classes of deep learning models, ensemble models, and hybrid models. Application domains include stock market, marketing, E-commerce, corporate banking, and cryptocurrency. Prisma method, a systematic literature review methodology is used to ensure the quality of the survey. The findings revealed that the trends are on advancement of hybrid models as more than 51% of the reviewed articles applied hybrid model. On the other hand, it is found that based on the RMSE accuracy metric, hybrid models had higher prediction accuracy than other algorithms. While it is expected the trends go toward the advancements of deep learning models.

scholarly journals Barriers and facilitators to data quality of electronic health records used for clinical research in China: a qualitative study

BMJ Open ◽  
2019 ◽  
Vol 9 (7) ◽  
pp. e029314 ◽  
Author(s):  
Kaiwen Ni ◽  
Hongling Chu ◽  
Lin Zeng ◽  
Nan Li ◽  
Yiming Zhao
Keyword(s):  
Qualitative Study ◽  
Electronic Health Records ◽  
Clinical Research ◽  
Data Quality ◽  
Healthcare Professionals ◽  
Barriers And Facilitators ◽  
Health Records ◽  
Improve Data Quality ◽  
Electronic Health

ObjectivesThere is an increasing trend in the use of electronic health records (EHRs) for clinical research. However, more knowledge is needed on how to assure and improve data quality. This study aimed to explore healthcare professionals’ experiences and perceptions of barriers and facilitators of data quality of EHR-based studies in the Chinese context.SettingFour tertiary hospitals in Beijing, China.ParticipantsNineteen healthcare professionals with experience in using EHR data for clinical research participated in the study.MethodsA qualitative study based on face-to-face semistructured interviews was conducted from March to July 2018. The interviews were audiorecorded and transcribed verbatim. Data analysis was performed using the inductive thematic analysis approach.ResultsThe main themes included factors related to healthcare systems, clinical documentation, EHR systems and researchers. The perceived barriers to data quality included heavy workload, staff rotations, lack of detailed information for specific research, variations in terminology, limited retrieval capabilities, large amounts of unstructured data, challenges with patient identification and matching, problems with data extraction and unfamiliar with data quality assessment. To improve data quality, suggestions from participants included: better staff training, providing monetary incentives, performing daily data verification, improving software functionality and coding structures as well as enhancing multidisciplinary cooperation.ConclusionsThese results provide a basis to begin to address current barriers and ultimately to improve validity and generalisability of research findings in China.

scholarly journals XAS Data Preprocessing of Nanocatalysts for Machine Learning Applications

Materials ◽  
2021 ◽  
Vol 14 (24) ◽  
pp. 7884
Author(s):  
Oleg O. Kartashov ◽  
Andrey V. Chernov ◽  
Dmitry S. Polyanichenko ◽  
Maria A. Butakova
Keyword(s):  
Machine Learning ◽  
Experimental Data ◽  
Synchrotron Radiation ◽  
Functional Materials ◽  
Data Preprocessing ◽  
Quality Data ◽  
X Ray ◽  
Physical Experiments ◽  
X Ray Absorption

Innovative development in the energy and chemical industries is mainly dependent on advances in the accelerated design and development of new functional materials. The success of research in new nanocatalysts mainly relies on modern techniques and approaches for their precise characterization. The existing methods of experimental characterization of nanocatalysts, which make it possible to assess the possibility of using these materials in specific chemical reactions or applications, generate significant amounts of heterogeneous data. The acceleration of new functional materials, including nanocatalysts, directly depends on the speed and quality of extracting hidden dependencies and knowledge from the obtained experimental data. Usually, such experiments involve different characterization techniques and different types of X-ray absorption spectroscopy (XAS) too. Using the machine learning (ML) methods based on XAS data, we can study and predict the atomic-scale structure and another bunch of parameters for the nanocatalyst efficiently. However, before using any ML model, it is necessary to make sure that the XAS raw experimental data is properly pre-processed, cleared, and prepared for ML application. Usually, the XAS preprocessing stage is vaguely presented in scientific studies, and the main efforts of researchers are devoted to the ML description and implementation stage. However, the quality of the input data influences the quality of ML analysis and the prediction results used in the future. This paper fills the gap between the stage of obtaining XAS data from synchrotron facilities and the stage of using and customizing various ML analysis and prediction models. We aimed this study to develop automated tools for the preprocessing and presentation of data from physical experiments and the creation of deposited datasets on the basis of the example of studying palladium-based nanocatalysts using synchrotron radiation facilities. During the study, methods of preliminary processing of XAS data were considered, which can be conditionally divided into X-ray absorption near edge structure (XANES) and extended X-ray absorption fine structure (EXAFS). This paper proposes a software toolkit that implements data preprocessing scenarios in the form of a single pipeline. The main preprocessing methods used in this study proposed are principal component analysis (PCA); z-score normalization; the interquartile method for eliminating outliers in the data; as well as the k-means machine learning method, which makes it possible to clarify the phase of the studied material sample by clustering feature vectors of experiments. Among the results of this study, one should also highlight the obtained deposited datasets of physical experiments on palladium-based nanocatalysts using synchrotron radiation. This will allow for further high-quality data mining to extract new knowledge about materials using artificial intelligence methods and machine learning models, and will ensure the smooth dissemination of these datasets to researchers and their reuse.

scholarly journals Data Science in Economics: Comprehensive Review of Advanced Machine Learning and Deep Learning Methods

2020 ◽  
Author(s):  
Saeed Nosratabadi ◽  
Amir Mosavi ◽  
Puhong Duan ◽  
Pedram Ghamisi ◽  
Filip Ferdinand ◽  
...  
Keyword(s):  
Machine Learning ◽  
Deep Learning ◽  
Prediction Accuracy ◽  
Data Science ◽  
State Of The Art ◽  
Hybrid Models ◽  
The Other ◽  
Learning Models ◽  
Comprehensive Review

This paper provides the state of the art of data science in economics. Through a novel taxonomy of applications and methods advances in data science are investigated. The data science advances are investigated in three individual classes of deep learning models, ensemble models, and hybrid models. Application domains include stock market, marketing, E-commerce, corporate banking, and cryptocurrency. Prisma method, a systematic literature review methodology is used to ensure the quality of the survey. The findings revealed that the trends are on advancement of hybrid models as more than 51% of the reviewed articles applied hybrid model. On the other hand, it is found that based on the RMSE accuracy metric, hybrid models had higher prediction accuracy than other algorithms. While it is expected the trends go toward the advancements of deep learning models.

Sign in / Sign up

Export Citation Format

Share Document