Construction of a quality model for machine learning systems

AbstractNowadays, systems containing components based on machine learning (ML) methods are becoming more widespread. In order to ensure the intended behavior of a software system, there are standards that define necessary qualities of the system and its components (such as ISO/IEC 25010). Due to the different nature of ML, we have to re-interpret existing qualities for ML systems or add new ones (such as trustworthiness). We have to be very precise about which quality property is relevant for which entity of interest (such as completeness of training data or correctness of trained model), and how to objectively evaluate adherence to quality requirements. In this article, we present how to systematically construct quality models for ML systems based on an industrial use case. This quality model enables practitioners to specify and assess qualities for ML systems objectively. In addition to the overall construction process described, the main outcomes include a meta-model for specifying quality models for ML systems, reference elements regarding relevant views, entities, quality properties, and measures for ML systems based on existing research, an example instantiation of a quality model for a concrete industrial use case, and lessons learned from applying the construction process. We found that it is crucial to follow a systematic process in order to come up with measurable quality properties that can be evaluated in practice. In the future, we want to learn how the term quality differs between different types of ML systems and come up with reference quality models for evaluating qualities of ML systems.

Download Full-text

Broadening volcanic eruption forecasting using transfer machine learning

10.5194/egusphere-egu21-970 ◽

2021 ◽

Author(s):

David Dempsey ◽

Shane Cronin ◽

Andreas Kempa-Liehr ◽

Martin Letourneur

Keyword(s):

Machine Learning ◽

Seismic Station ◽

Feature Space ◽

Forecast Model ◽

Linear Interpolation ◽

Lessons Learned ◽

Training Data ◽

Single Station ◽

Data Driven Approach ◽

Model Training

Sudden steam-driven eruptions at tourist volcanoes were the cause of 63 deaths at Mt Ontake (Japan) in 2014, and 22 deaths at Whakaari (New Zealand) in 2019. Warning systems that can anticipate these eruptions could provide crucial hours for evacuation or sheltering but these require reliable forecasting. Recently, machine learning has been used to extract eruption precursors from observational data and train forecasting models. However, a weakness of this data-driven approach is its reliance on long observational records that span multiple eruptions. As many volcano datasets may only record one or no eruptions, there is a need to extend these techniques to data-poor locales.Transfer machine learning is one approach for generalising lessons learned at data-rich volcanoes and applying them to data-poor ones. Here, we tackle two problems: (1) generalising time series features between seismic stations at Whakaari to address recording gaps, and (2) training a forecasting model for Mt Ruapehu augmented using data from Whakaari. This required that we standardise data records at different stations for direct comparisons, devise an interpolation scheme to fill in missing eruption data, and combine volcano-specific feature matrices prior to model training.We trained a forecast model for Whakaari using tremor data from three eruptions recorded at one seismic station (WSRZ) and augmented by data from two other eruptions recorded at a second station (WIZ). First, the training data from both stations were standardised to a unit normal distribution in log space. Then, linear interpolation in feature space was used to infer missing eruption features at WSRZ. Under pseudo-prospective testing, the augmented model had similar forecasting skill to one trained using all five eruptions recorded at a single station (WIZ). However, extending this approach to Ruapehu, we saw reduced performance indicating that more work is needed in standardisation and feature selection.

Download Full-text

A User-Centered Approach to Gamify the Manual Creation of Training Data for Machine Learning

i-com ◽

10.1515/icom-2020-0030 ◽

2021 ◽

Vol 20 (1) ◽

pp. 33-48

Author(s):

Sarah Alaghbari ◽

Annett Mitschick ◽

Gregor Blichmann ◽

Martin Voigt ◽

Raimund Dachselt

Keyword(s):

Machine Learning ◽

Real Life ◽

Training Data ◽

Use Case ◽

Target User ◽

Game Context ◽

Data Objects ◽

Game Elements ◽

Annotation Use ◽

User Centered

Abstract The development of artificial intelligence, e. g. for Computer Vision, through supervised learning requires the input of large amounts of annotated or labeled data objects as training data. Usually, the creation of high-quality training data is done manually which can be repetitive and tiring. Gamification, the use of game elements in a non-game context, is one method to make such tedious tasks more interesting. We propose a multi-step process for gamifying the manual creation of training data for machine learning purposes. In this article, we give an overview of related concepts and existing implementations and present a user-centered approach for a real-life use case. Based on a survey within the target user group we identified annotation use cases and dominant player characteristics. The results served as a foundation for designing the gamification concepts which were then discussed with the participants. The final concept includes levels of increasing difficulty, tutorials, progress indicators and a narrative built around a robot character which at the same time is a user assistant. The implemented prototype is an extension of an existing annotation tool at an AI product company and serves as a basis for further observations.

Download Full-text

Curating Archival Anatomic Pathology Material For Machine Learning Algorithm Development

American Journal of Clinical Pathology ◽

10.1093/ajcp/aqaa161.272 ◽

2020 ◽

Vol 154 (Supplement_1) ◽

pp. S124-S125

Author(s):

A Collins ◽

A Norgan ◽

J J Garcia

Keyword(s):

Machine Learning ◽

Cell Carcinoma ◽

Learning Algorithm ◽

Lessons Learned ◽

Machine Learning Algorithms ◽

Training Data ◽

Machine Learning Algorithm ◽

Anatomic Site ◽

Algorithm Development ◽

Anatomic Pathology

Abstract Introduction/Objective Advances in whole slide imaging have enabled the application of machine learning algorithms to anatomic pathology. In the current state, the development of accurate algorithms requires robust training data with correctly assigned diagnostic and classification labels. Increasingly, institutions have looked to their archival slides as a source of “ground truth” for algorithm development. However, the curation and use of archival data poses several challenges. Here, we share lessons learned from reviewing head and neck pathology consult cases spanning a 10- year period at Mayo Clinic Rochester. Methods Archived surgical pathology slides from 2,590 consult cases were reviewed. Clinical and demographic information was recorded for each case, including surgical date, surgical procedure, anatomic site, age, gender and diagnosis. Cases were excluded from the curated archive if there was insufficient volume or quality of tissue to render a specific diagnosis (141 cases, 5.6%). Slides with a range of tissue size and quality, from numerable laboratories were included in the curated archive. Selected cases were collated by anatomic site: ear, gnathic, larynx, nasopharynx, neck, oral cavity, oropharynx, salivary gland and sinonasal tract. Results Common diagnostic reconciliations (115 cases, 4.4%) fell within the following categories: (1) novel entities (59 cases, 2.3%), including biphenotypic sinonasal sarcoma and clear cell carcinoma; (2) novel classifications (21 cases, 0.8%), as seen in HPV-related oropharyngeal squamous cell carcinoma and polymorphous adenocarcinoma; and (3) novel grading schema (35 cases, 1.4%), as seen in keratinizing dysplasia and oropharyngeal malignancies. Conclusion Several nuances emerged in the process of reviewing slides, highlighting the need for continual amendment of any machine learning dataset over time. Curating anatomic pathology cases for machine learning algorithm development requires the recognition of emerging entities, with re-classification and re-grading as needed.

Download Full-text

Hierarchy-based semantic embeddings for single-valued & multi-valued categorical variables

Journal of Intelligent Information Systems ◽

10.1007/s10844-021-00693-2 ◽

2021 ◽

Author(s):

Summaya Mumtaz ◽

Martin Giese

Keyword(s):

Machine Learning ◽

Similarity Measure ◽

Domain Knowledge ◽

Similarity Measures ◽

Training Data ◽

Mixed Data ◽

Categorical Variables ◽

Use Case ◽

Data Types ◽

Semantic Similarity Measure

AbstractIn low-resource domains, it is challenging to achieve good performance using existing machine learning methods due to a lack of training data and mixed data types (numeric and categorical). In particular, categorical variables with high cardinality pose a challenge to machine learning tasks such as classification and regression because training requires sufficiently many data points for the possible values of each variable. Since interpolation is not possible, nothing can be learned for values not seen in the training set. This paper presents a method that uses prior knowledge of the application domain to support machine learning in cases with insufficient data. We propose to address this challenge by using embeddings for categorical variables that are based on an explicit representation of domain knowledge (KR), namely a hierarchy of concepts. Our approach is to 1. define a semantic similarity measure between categories, based on the hierarchy—we propose a purely hierarchy-based measure, but other similarity measures from the literature can be used—and 2. use that similarity measure to define a modified one-hot encoding. We propose two embedding schemes for single-valued and multi-valued categorical data. We perform experiments on three different use cases. We first compare existing similarity approaches with our approach on a word pair similarity use case. This is followed by creating word embeddings using different similarity approaches. A comparison with existing methods such as Google, Word2Vec and GloVe embeddings on several benchmarks shows better performance on concept categorisation tasks when using knowledge-based embeddings. The third use case uses a medical dataset to compare the performance of semantic-based embeddings and standard binary encodings. Significant improvement in performance of the downstream classification tasks is achieved by using semantic information.

Download Full-text

Orchestrating Heterogeneous Devices and AI Services as Virtual Sensors for Secure Cloud-Based IoT Applications

Sensors ◽

10.3390/s21227509 ◽

2021 ◽

Vol 21 (22) ◽

pp. 7509

Author(s):

Sebastian Alberternst ◽

Alexander Anisimov ◽

Andre Antakli ◽

Benjamin Duppe ◽

Hilko Hoffmann ◽

...

Keyword(s):

Machine Learning ◽

Assisted Living ◽

Training Data ◽

Use Case ◽

Learning Approaches ◽

Virtual Sensors ◽

Agent Based ◽

Semantic Level ◽

Heterogeneous Devices ◽

Cloud Environments

The concept of the cloud-to-thing continuum addresses advancements made possible by the widespread adoption of cloud, edge, and IoT resources. It opens the possibility of combining classical symbolic AI with advanced machine learning approaches in a meaningful way. In this paper, we present a thing registry and an agent-based orchestration framework, which we combine to support semantic orchestration of IoT use cases across several federated cloud environments. We use the concept of virtual sensors based on machine learning (ML) services as abstraction, mediating between the instance level and the semantic level. We present examples of virtual sensors based on ML models for activity recognition and describe an approach to remedy the problem of missing or scarce training data. We illustrate the approach with a use case from an assisted living scenario.

Download Full-text

Scalable Approach to High Coverages on Oxides via Iterative Training of a Machine-Learning Algorithm

10.26434/chemrxiv.10288514.v1 ◽

2019 ◽

Author(s):

Andrew Medford ◽

Shengchun Yang ◽

Fuzhu Liu

Keyword(s):

Machine Learning ◽

Chemical Potential ◽

Learning Algorithm ◽

Absolute Error ◽

Low Energy ◽

Training Data ◽

High Coverage ◽

Metal Compounds ◽

Adsorption Energies ◽

The Stability

Understanding the interaction of multiple types of adsorbate molecules on solid surfaces is crucial to establishing the stability of catalysts under various chemical environments. Computational studies on the high coverage and mixed coverages of reaction intermediates are still challenging, especially for transition-metal compounds. In this work, we present a framework to predict differential adsorption energies and identify low-energy structures under high- and mixed-adsorbate coverages on oxide materials. The approach uses Gaussian process machine-learning models with quantified uncertainty in conjunction with an iterative training algorithm to actively identify the training set. The framework is demonstrated for the mixed adsorption of CHx, NHx and OHx species on the oxygen vacancy and pristine rutile TiO2(110) surface sites. The results indicate that the proposed algorithm is highly efficient at identifying the most valuable training data, and is able to predict differential adsorption energies with a mean absolute error of ~0.3 eV based on <25% of the total DFT data. The algorithm is also used to identify 76% of the low-energy structures based on <30% of the total DFT data, enabling construction of surface phase diagrams that account for high and mixed coverage as a function of the chemical potential of C, H, O, and N. Furthermore, the computational scaling indicates the algorithm scales nearly linearly (N1.12) as the number of adsorbates increases. This framework can be directly extended to metals, metal oxides, and other materials, providing a practical route toward the investigation of the behavior of catalysts under high-coverage conditions.

Download Full-text

Optimization of Diabetes Training DATA using Machine Learning Algorithms

International Journal of Computer Sciences and Engineering ◽

10.26438/ijcse/v6i2.283286 ◽

2018 ◽

Vol 6 (2) ◽

pp. 283-286

Author(s):

M. Samba Siva Rao ◽

◽

M.Yaswanth . ◽

K. Raghavendra Swamy ◽

◽

...

Keyword(s):

Machine Learning ◽

Learning Algorithms ◽

Machine Learning Algorithms ◽

Training Data

Download Full-text

River Quality Models for Consent Setting in England and Wales

Water Science & Technology ◽

10.2166/wst.1989.0303 ◽

1989 ◽

Vol 21 (8-9) ◽

pp. 1015-1024 ◽

Cited By ~ 4

Author(s):

C. P. Crockett ◽

R. W. Crabtree ◽

I. D. Cluckie

Keyword(s):

River System ◽

Quality Model ◽

Catchment Scale ◽

Quality Models ◽

England And Wales ◽

Future Developments ◽

Statistical Framework ◽

River Quality ◽

National Approach

In England and Wales the placing of effluent discharge consents within a statistical framework has led to the development of a new hybrid type of river quality model. Such catchment scale consent models have a stochastic component for the generation of model inputs and a deterministic component to route them through the river system. This paper reviews and compares the existing approaches for consent modelling used by various Water Authorities. A number of possible future developments are suggested including the potential need for a national approach to the review and setting of long term consents.

Download Full-text

Comparative Analysis of Machine Learning Techniques Using Predictive Modeling

Recent Advances in Computer Science and Communications ◽

10.2174/2666255813999200904164539 ◽

2020 ◽

Vol 13 ◽

Author(s):

Ritu Khandelwal ◽

Hemlata Goyal ◽

Rajveer Singh Shekhawat

Keyword(s):

Machine Learning ◽

Comparative Analysis ◽

Data Science ◽

Training Data ◽

Machine Learning Techniques ◽

Future Trends ◽

Data Set ◽

Learning Stage ◽

Learning Techniques ◽

Different Types

Introduction: Machine learning is an intelligent technology that works as a bridge between businesses and data science. With the involvement of data science, the business goal focuses on findings to get valuable insights on available data. The large part of Indian Cinema is Bollywood which is a multi-million dollar industry. This paper attempts to predict whether the upcoming Bollywood Movie would be Blockbuster, Superhit, Hit, Average or Flop. For this Machine Learning techniques (classification and prediction) will be applied. To make classifier or prediction model first step is the learning stage in which we need to give the training data set to train the model by applying some technique or algorithm and after that different rules are generated which helps to make a model and predict future trends in different types of organizations. Methods: All the techniques related to classification and Prediction such as Support Vector Machine(SVM), Random Forest, Decision Tree, Naïve Bayes, Logistic Regression, Adaboost, and KNN will be applied and try to find out efficient and effective results. All these functionalities can be applied with GUI Based workflows available with various categories such as data, Visualize, Model, and Evaluate. Result: To make classifier or prediction model first step is learning stage in which we need to give the training data set to train the model by applying some technique or algorithm and after that different rules are generated which helps to make a model and predict future trends in different types of organizations Conclusion: This paper focuses on Comparative Analysis that would be performed based on different parameters such as Accuracy, Confusion Matrix to identify the best possible model for predicting the movie Success. By using Advertisement Propaganda, they can plan for the best time to release the movie according to the predicted success rate to gain higher benefits. Discussion: Data Mining is the process of discovering different patterns from large data sets and from that various relationships are also discovered to solve various problems that come in business and helps to predict the forthcoming trends. This Prediction can help Production Houses for Advertisement Propaganda and also they can plan their costs and by assuring these factors they can make the movie more profitable.

Download Full-text

Multilayer Soil Moisture Mapping at a Regional Scale from Multisource Data via a Machine Learning Method

Remote Sensing ◽

10.3390/rs11030284 ◽

2019 ◽

Vol 11 (3) ◽

pp. 284 ◽

Cited By ~ 1

Author(s):

Linglin Zeng ◽

Shun Hu ◽

Daxiang Xiang ◽

Xiang Zhang ◽

Deren Li ◽

...

Keyword(s):

Machine Learning ◽

Soil Moisture ◽

Regional Scale ◽

Remotely Sensed ◽

Temporal Variations ◽

Training Data ◽

Estimation Accuracy ◽

Learning Approaches ◽

Remotely Sensed Data ◽

Deep Soil

Soil moisture mapping at a regional scale is commonplace since these data are required in many applications, such as hydrological and agricultural analyses. The use of remotely sensed data for the estimation of deep soil moisture at a regional scale has received far less emphasis. The objective of this study was to map the 500-m, 8-day average and daily soil moisture at different soil depths in Oklahoma from remotely sensed and ground-measured data using the random forest (RF) method, which is one of the machine-learning approaches. In order to investigate the estimation accuracy of the RF method at both a spatial and a temporal scale, two independent soil moisture estimation experiments were conducted using data from 2010 to 2014: a year-to-year experiment (with a root mean square error (RMSE) ranging from 0.038 to 0.050 m3/m3) and a station-to-station experiment (with an RMSE ranging from 0.044 to 0.057 m3/m3). Then, the data requirements, importance factors, and spatial and temporal variations in estimation accuracy were discussed based on the results using the training data selected by iterated random sampling. The highly accurate estimations of both the surface and the deep soil moisture for the study area reveal the potential of RF methods when mapping soil moisture at a regional scale, especially when considering the high heterogeneity of land-cover types and topography in the study area.

Download Full-text