Curator: A No-Code Self-Supervised Learning and Active Labeling Tool to Create Labeled Image Datasets from Petabyte-Scale Imagery

2021 ◽  
Author(s):  
Rudy Venguswamy ◽  
Mike Levy ◽  
Anirudh Koul ◽  
Satyarth Praveen ◽  
Tarun Narayanan ◽  
...  
Keyword(s):  
Machine Learning ◽  
Active Learning ◽  
Open Source ◽  
Forest Fires ◽  
Seed Set ◽  
Training Data ◽  
Training Dataset ◽  
Reference Image ◽  
Query Image ◽  
Real World Datasets

<p>Machine learning modeling for Earth events at NASA is often limited by the availability of labeled examples. For example, training classifiers for forest fires or oil spills from satellite imagery requires curating a massive and diverse dataset of example forest fires, a tedious multi-month effort requiring careful review of over 196.9 million square miles of data per day for 20 years. While such images might exist in abundance within 40 petabytes of unlabeled satellite data, finding these positive examples to include in a training dataset for a machine learning model is extremely time-consuming and requires researchers to "hunt" for positive examples, like finding a needle in a haystack. </p><p>We present a no-code open-source tool, Curator, whose goal is to minimize the amount of human manual image labeling needed to achieve a state of the art classifier. The pipeline, purpose-built to take advantage of the massive amount of unlabeled images, consists of (1) self-supervision training to convert unlabeled images into meaningful representations, (2) search-by-example to collect a seed set of images, (3) human-in-the-loop active learning to iteratively ask for labels on uncertain examples and train on them. </p><p>In step 1, a model capable of representing unlabeled images meaningfully is trained with a self-supervised algorithm (like SimCLR) on a random subset of the dataset (that conforms to researchers’ specified “training budget.”). Since real-world datasets are often imbalanced leading to suboptimal models, the initial model is used to generate embeddings on the entire dataset. Then, images with equidistant embeddings are sampled. This iterative training and resampling strategy improves both balanced training data and models every iteration. In step 2, researchers supply an example image of interest, and the output embeddings generated from this image are used to find other images with embeddings near the reference image’s embedding in euclidean space (hence similar looking images to the query image). These proposed candidate images contain a higher density of positive examples and are annotated manually as a seed set. In step 3, the seed labels are used to train a classifier to identify more candidate images for human inspection with active learning. Each classification training loop, candidate images for labeling are sampled from the larger unlabeled dataset based on the images that the model is most uncertain about (p ≈ 0.5).</p><p>Curator is released as an open-source package built on PyTorch-Lightning. The pipeline uses GPU-based transforms from the NVIDIA-Dali package for augmentation, leading to a 5-10x speed up in self-supervised training and is run from the command line.</p><p>By iteratively training a self-supervised model and a classifier in tandem with human manual annotation, this pipeline is able to unearth more positive examples from severely imbalanced datasets which were previously untrainable with self-supervision algorithms. In applications such as detecting wildfires, atmospheric dust, or turning outward with telescopic surveys, increasing the number of positive candidates presented to humans for manual inspection increases the efficacy of classifiers and multiplies the efficiency of researchers’ data curation efforts.</p>

scholarly journals Zero Initialized Active Learning with Spectral Clustering using Hungarian Method

2021 ◽  
Vol 25 (2) ◽  
pp. 401-419
Author(s):  
Dávid Papp
Keyword(s):  
Machine Learning ◽  
Active Learning ◽  
Spectral Clustering ◽  
Clustering Algorithm ◽  
Learning System ◽  
Training Data ◽  
Supervised Machine Learning ◽  
Training Dataset ◽  
Generative Adversarial Networks ◽  
Class Membership

Supervised machine learning tasks often require a large number of labeled training data to set up a model, and then prediction - for example the classification - is carried out based on this model. Nowadays tremendous amount of data is available on the web or in data warehouses, although only a portion of those data is annotated and the labeling process can be tedious, expensive and time consuming. Active learning tries to overcome this problem by reducing the labeling cost through allowing the learning system to iteratively select the data from which it learns. In special case of active learning, the process starts from zero initialized scenario, where the labeled training dataset is empty, and therefore only unsupervised methods can be performed. In this paper a novel query strategy framework is presented for this problem, called Clustering Based Balanced Sampling Framework (CBBSF), which is not only select the initial labeled training dataset, but uniformly selects the items among the categories to get a balanced labeled training dataset. The framework includes an assignment technique to implicitly determine the class membership probabilities. Assignment solution is updated during CBBSF iterations, hence it simulates supervised machine learning more accurately as the process progresses. The proposed Spectral Clustering Based Sampling (SCBS) query startegy realizes the CBBSF framework, and therefore it is applicable in the special zero initialized situation. This selection approach uses ClusterGAN (Clustering using Generative Adversarial Networks) integrated in the spectral clustering algorithm and then it selects an unlabeled instance depending on the class membership probabilities. Global and local versions of SCBS were developed, furthermore, most confident and minimal entropy measures were calculated, thus four different SCBS variants were examined in total. Experimental evaluation was conducted on the MNIST dataset, and the results showed that SCBS outperforms the state-of-the-art zero initialized active learning query strategies.

Headnote Prediction Using Machine Learning

2021 ◽  
Vol 18 (5) ◽  
Author(s):  
Sarmad Mahar ◽  
Sahar Zafar ◽  
Kamran Nishat
Keyword(s):  
Machine Learning ◽  
Feature Extraction ◽  
Active Learning ◽  
Text Classification ◽  
Extraction Methods ◽  
Text Summarization ◽  
Training Data ◽  
Second Step ◽  
Support Vector ◽  
Classification Algorithms

Headnotes are the precise explanation and summary of legal points in an issued judgment. Law journals hire experienced lawyers to write these headnotes. These headnotes help the reader quickly determine the issue discussed in the case. Headnotes comprise two parts. The first part comprises the topic discussed in the judgment, and the second part contains a summary of that judgment. In this thesis, we design, develop and evaluate headnote prediction using machine learning, without involving human involvement. We divided this task into a two steps process. In the first step, we predict law points used in the judgment by using text classification algorithms. The second step generates a summary of the judgment using text summarization techniques. To achieve this task, we created a Databank by extracting data from different law sources in Pakistan. We labelled training data generated based on Pakistan law websites. We tested different feature extraction methods on judiciary data to improve our system. Using these feature extraction methods, we developed a dictionary of terminology for ease of reference and utility. Our approach achieves 65% accuracy by using Linear Support Vector Classification with tri-gram and without stemmer. Using active learning our system can continuously improve the accuracy with the increased labelled examples provided by the users of the system.

scholarly journals Zero-Shot Feature Selection via Transferring Supervised Knowledge

2021 ◽  
Vol 17 (2) ◽  
pp. 1-20
Author(s):  
Zheng Wang ◽  
Qiao Wang ◽  
Tingzhang Zhao ◽  
Chaokun Wang ◽  
Xiaojun Ye
Keyword(s):  
Machine Learning ◽  
Feature Selection ◽  
Dimensionality Reduction ◽  
Real World ◽  
Rapid Growth ◽  
Learning Systems ◽  
Training Data ◽  
Effective Technique ◽  
Supervised Methods ◽  
Real World Datasets

Feature selection, an effective technique for dimensionality reduction, plays an important role in many machine learning systems. Supervised knowledge can significantly improve the performance. However, faced with the rapid growth of newly emerging concepts, existing supervised methods might easily suffer from the scarcity and validity of labeled data for training. In this paper, the authors study the problem of zero-shot feature selection (i.e., building a feature selection model that generalizes well to “unseen” concepts with limited training data of “seen” concepts). Specifically, they adopt class-semantic descriptions (i.e., attributes) as supervision for feature selection, so as to utilize the supervised knowledge transferred from the seen concepts. For more reliable discriminative features, they further propose the center-characteristic loss which encourages the selected features to capture the central characteristics of seen concepts. Extensive experiments conducted on various real-world datasets demonstrate the effectiveness of the method.

scholarly journals HYBRID ACQUISITION OF HIGH QUALITY TRAINING DATA FOR SEMANTIC SEGMENTATION OF 3D POINT CLOUDS USING CROWD-BASED ACTIVE LEARNING

2020 ◽  
Vol V-2-2020 ◽  
pp. 501-508
Author(s):  
M. Kölle ◽  
V. Walter ◽  
S. Schmohl ◽  
U. Soergel
Keyword(s):  
Active Learning ◽  
Semantic Segmentation ◽  
Point Clouds ◽  
Human Interaction ◽  
Training Data ◽  
Semantic Interpretation ◽  
Training Dataset ◽  
3D Point Clouds ◽  
Semantic Labeling ◽  
3D Cnn

Abstract. Automated semantic interpretation of 3D point clouds is crucial for many tasks in the domain of geospatial data analysis. For this purpose, labeled training data is required, which has often to be provided manually by experts. One approach to minimize effort in terms of costs of human interaction is Active Learning (AL). The aim is to process only the subset of an unlabeled dataset that is particularly helpful with respect to class separation. Here a machine identifies informative instances which are then labeled by humans, thereby increasing the performance of the machine. In order to completely avoid involvement of an expert, this time-consuming annotation can be resolved via crowdsourcing. Therefore, we propose an approach combining AL with paid crowdsourcing. Although incorporating human interaction, our method can run fully automatically, so that only an unlabeled dataset and a fixed financial budget for the payment of the crowdworkers need to be provided. We conduct multiple iteration steps of the AL process on the ISPRS Vaihingen 3D Semantic Labeling benchmark dataset (V3D) and especially evaluate the performance of the crowd when labeling 3D points. We prove our concept by using labels derived from our crowd-based AL method for classifying the test dataset. The analysis outlines that by labeling only 0:4% of the training dataset by the crowd and spending less than 145 $, both our trained Random Forest and sparse 3D CNN classifier differ in Overall Accuracy by less than 3 percentage points compared to the same classifiers trained on the complete V3D training set.

Combining Self-supervised Learning and Active Learning for Disfluency Detection

2022 ◽  
Vol 21 (3) ◽  
pp. 1-25
Author(s):  
Shaolei Wang ◽  
Zhongyuan Wang ◽  
Wanxiang Che ◽  
Sendong Zhao ◽  
Ting Liu
Keyword(s):  
Neural Network ◽  
Active Learning ◽  
Supervised Learning ◽  
Large Scale ◽  
Training Data ◽  
Fine Tuning ◽  
Training Dataset ◽  
Performance Gap ◽  
Annotation Costs ◽  
Trained Neural Network

Spoken language is fundamentally different from the written language in that it contains frequent disfluencies or parts of an utterance that are corrected by the speaker. Disfluency detection (removing these disfluencies) is desirable to clean the input for use in downstream NLP tasks. Most existing approaches to disfluency detection heavily rely on human-annotated data, which is scarce and expensive to obtain in practice. To tackle the training data bottleneck, in this work, we investigate methods for combining self-supervised learning and active learning for disfluency detection. First, we construct large-scale pseudo training data by randomly adding or deleting words from unlabeled data and propose two self-supervised pre-training tasks: (i) a tagging task to detect the added noisy words and (ii) sentence classification to distinguish original sentences from grammatically incorrect sentences. We then combine these two tasks to jointly pre-train a neural network. The pre-trained neural network is then fine-tuned using human-annotated disfluency detection training data. The self-supervised learning method can capture task-special knowledge for disfluency detection and achieve better performance when fine-tuning on a small annotated dataset compared to other supervised methods. However, limited in that the pseudo training data are generated based on simple heuristics and cannot fully cover all the disfluency patterns, there is still a performance gap compared to the supervised models trained on the full training dataset. We further explore how to bridge the performance gap by integrating active learning during the fine-tuning process. Active learning strives to reduce annotation costs by choosing the most critical examples to label and can address the weakness of self-supervised learning with a small annotated dataset. We show that by combining self-supervised learning with active learning, our model is able to match state-of-the-art performance with just about 10% of the original training data on both the commonly used English Switchboard test set and a set of in-house annotated Chinese data.

scholarly journals A machine learning framework to determine geolocations from metagenomic profiling

2020 ◽  
Vol 15 (1) ◽  
Author(s):  
Lihong Huang ◽  
Canqiang Xu ◽  
Wenxian Yang ◽  
Rongshan Yu
Keyword(s):  
Machine Learning ◽  
Prediction Models ◽  
Geographic Origin ◽  
Training Data ◽  
Metagenomic Data ◽  
Training Dataset ◽  
Kriging Interpolation ◽  
Learning Framework ◽  
Testing Data ◽  
Microbial Samples

Abstract Background Studies on metagenomic data of environmental microbial samples found that microbial communities seem to be geolocation-specific, and the microbiome abundance profile can be a differentiating feature to identify samples’ geolocations. In this paper, we present a machine learning framework to determine the geolocations from metagenomics profiling of microbial samples. Results Our method was applied to the multi-source microbiome data from MetaSUB (The Metagenomics and Metadesign of Subways and Urban Biomes) International Consortium for the CAMDA 2019 Metagenomic Forensics Challenge (the Challenge). The goal of the Challenge is to predict the geographical origins of mystery samples by constructing microbiome fingerprints.First, we extracted features from metagenomic abundance profiles. We then randomly split the training data into training and validation sets and trained the prediction models on the training set. Prediction performance was evaluated on the validation set. By using logistic regression with L2 normalization, the prediction accuracy of the model reaches 86%, averaged over 100 random splits of training and validation datasets.The testing data consists of samples from cities that do not occur in the training data. To predict the “mystery” cities that are not sampled before for the testing data, we first defined biological coordinates for sampled cities based on the similarity of microbial samples from them. Then we performed affine transform on the map such that the distance between cities measures their biological difference rather than geographical distance. After that, we derived the probabilities of a given testing sample from unsampled cities based on its predicted probabilities on sampled cities using Kriging interpolation. Results show that this method can successfully assign high probabilities to the true cities-of-origin of testing samples. Conclusion Our framework shows good performance in predicting the geographic origin of metagenomic samples for cities where training data are available. Furthermore, we demonstrate the potential of the proposed method to predict metagenomic samples’ geolocations for samples from locations that are not in the training dataset.

scholarly journals Noise Removal Process from Label Classification using Machine Learning

2019 ◽  
Vol 8 (3) ◽  
pp. 172-175
Keyword(s):  
Machine Learning ◽  
Big Data ◽  
Supervised Learning ◽  
Noise Removal ◽  
Error Rates ◽  
Training Data ◽  
Learning Performance ◽  
Training Dataset ◽  
Noise Filtering ◽  
Label Noise

Text classification and clustering approach is essential for big data environments. In supervised learning applications many classification algorithms have been proposed. In the era of big data, a large volume of training data is available in many machine learning works. However, there is a possibility of mislabeled or unlabeled data that are not labeled properly. Some labels may be incorrect resulted in label noise which in turn regress learning performance of a classifier. A general approach to address label noise is to apply noise filtering techniques to identify and remove noise before learning. A range of noise filtering approaches have been developed to improve the classifiers performance. This paper proposes noise filtering approach in text data during the training phase. Many supervised learning algorithms generates high error rates due to noise in training dataset, our work eliminates such noise and provides accurate classification system.

scholarly journals Leveraging spatial textures, through machine learning, to identify aerosols and distinct cloud types from multispectral observations

2020 ◽  
Vol 13 (10) ◽  
pp. 5459-5480
Author(s):  
Willem J. Marais ◽  
Robert E. Holz ◽  
Jeffrey S. Reid ◽  
Rebecca M. Willett
Keyword(s):  
Machine Learning ◽  
Spectral Properties ◽  
Infrared Imaging ◽  
Spatial Information ◽  
Training Data ◽  
Machine Learning Techniques ◽  
Training Dataset ◽  
Orthogonal Polarization ◽  
Statistical Parameters ◽  
Moderate Resolution Imaging Spectroradiometer

Abstract. Current cloud and aerosol identification methods for multispectral radiometers, such as the Moderate Resolution Imaging Spectroradiometer (MODIS) and Visible Infrared Imaging Radiometer Suite (VIIRS), employ multichannel spectral tests on individual pixels (i.e., fields of view). The use of the spatial information in cloud and aerosol algorithms has been primarily through statistical parameters such as nonuniformity tests of surrounding pixels with cloud classification provided by the multispectral microphysical retrievals such as phase and cloud top height. With these methodologies there is uncertainty in identifying optically thick aerosols, since aerosols and clouds have similar spectral properties in coarse-spectral-resolution measurements. Furthermore, identifying clouds regimes (e.g., stratiform, cumuliform) from just spectral measurements is difficult, since low-altitude cloud regimes have similar spectral properties. Recent advances in computer vision using deep neural networks provide a new opportunity to better leverage the coherent spatial information in multispectral imagery. Using a combination of machine learning techniques combined with a new methodology to create the necessary training data, we demonstrate improvements in the discrimination between cloud and severe aerosols and an expanded capability to classify cloud types. The labeled training dataset was created from an adapted NASA Worldview platform that provides an efficient user interface to assemble a human-labeled database of cloud and aerosol types. The convolutional neural network (CNN) labeling accuracy of aerosols and cloud types was quantified using independent Cloud-Aerosol Lidar with Orthogonal Polarization (CALIOP) and MODIS cloud and aerosol products. By harnessing CNNs with a unique labeled dataset, we demonstrate the improvement of the identification of aerosols and distinct cloud types from MODIS and VIIRS images compared to a per-pixel spectral and standard deviation thresholding method. The paper concludes with case studies that compare the CNN methodology results with the MODIS cloud and aerosol products.

scholarly journals ROOFN3D: DEEP LEARNING TRAINING DATA FOR 3D BUILDING RECONSTRUCTION

2018 ◽  
Vol XLII-2 ◽  
pp. 1191-1198 ◽  
Author(s):  
A. Wichmann ◽  
A. Agoub ◽  
M. Kada
Keyword(s):  
Machine Learning ◽  
Deep Learning ◽  
Point Cloud ◽  
Geometric Model ◽  
Training Data ◽  
Computer Hardware ◽  
Training Dataset ◽  
3D Point Cloud ◽  
Learning Methods ◽  
Building Reconstruction

Machine learning methods have gained in importance through the latest development of artificial intelligence and computer hardware. Particularly approaches based on deep learning have shown that they are able to provide state-of-the-art results for various tasks. However, the direct application of deep learning methods to improve the results of 3D building reconstruction is often not possible due, for example, to the lack of suitable training data. To address this issue, we present RoofN3D which provides a new 3D point cloud training dataset that can be used to train machine learning models for different tasks in the context of 3D building reconstruction. It can be used, among others, to train semantic segmentation networks or to learn the structure of buildings and the geometric model construction. Further details about RoofN3D and the developed data preparation framework, which enables the automatic derivation of training data, are described in this paper. Furthermore, we provide an overview of other available 3D point cloud training data and approaches from current literature in which solutions for the application of deep learning to unstructured and not gridded 3D point cloud data are presented.

A Novel Approach for Colorization of a Grayscale Image using Soft Computing Techniques

2017 ◽  
Vol 8 (4) ◽  
pp. 19-43 ◽  
Author(s):  
Abul Hasnat ◽  
Santanu Halder ◽  
Debotosh Bhattacharjee ◽  
Mita Nasipuri
Keyword(s):  
Color Image ◽  
Digital Camera ◽  
Image Database ◽  
Turing Test ◽  
Training Dataset ◽  
Reference Image ◽  
Grayscale Image ◽  
Query Image ◽  
Novel Approach ◽  
Soft Computing Techniques

Colorization of grayscale image is a process to convert a grayscale image into a color one. Few research works reported in literature on this but there is hardly any generalized method that successfully colorizes all types of grayscale image. This study proposes a novel grayscale image colorization method using a reference color image. It takes the grayscale image and the type of the query image as input. First, it selects reference image from color image database using histogram index of the query image and histogram index of luminance channel of color images of respective type. Once the reference image is selected, four features are extracted for each pixel of the luminance channel of the reference image. These extracted features as input and chrominance blue(Cb) value as target value forms the training dataset for Cb channel. Similarly training dataset for chrominance red(Cr) channel is also formed. These extracted features of the reference image and associated chrominance values are used to train two artificial neural network(ANN)- one for Cb and one for Cr channel. Then, for each pixel of the of query image, same four features are extracted and used as input to the trained ANN to predict the chrominance values of the query image. Thus predicted chrominance values along with the original luminance values of the query image are used to construct the colorized image. The experiment has been conducted on images collected from different standard image database i.e. FRAV2D, UCID.v2 and images captured using standard digital camera etc. These images are initially converted into grayscale images and then the colorization method was applied. For performance evaluation, PSNR between the original color image and newly colorized image is calculated. PSNR shows that the proposed method better colorizes than the recently reported methods in the literature. Beside this, “Colorization Turing test” was conducted asking human subject to choose the image (closer to the original color image) among the colorized images using proposed algorithm and recently reported methods. In 80% of cases colorized images using the proposed method got selected.

Sign in / Sign up

Export Citation Format

Share Document