scholarly journals Assessing Uncertainty in LULC Classification Accuracy by Using Bootstrap Resampling

2016 ◽  
Author(s):  
Lin-Hsuan Hsiao ◽  
Ke-sheng Cheng
Keyword(s):  
Land Cover ◽  
Classification Accuracy ◽  
Training Data ◽  
Data Sets ◽  
Bootstrap Resampling ◽  
Chi Square ◽  
Data Set ◽  
Classification Uncertainty ◽  
Training Samples ◽  
Class Probability

Supervised land-use/land-cover (LULC) classifications are typically conducted using class assignment rules derived from a set of multiclass training samples. Consequently, classification accuracy varies with the training data set and is thus associated with uncertainty. In this study, we propose a bootstrap resampling and reclassification approach that can be applied for assessing not only the uncertainty in classification results of the bootstrap-training data sets, but also the classification uncertainty of individual pixels in the study area. Two measures of pixel-specific classification uncertainty, namely the maximum class probability and Shannon entropy, were derived from the class probability vector of individual pixels and used for the identification of unclassified pixels. Unclassified pixels that are identified using the traditional chi-square threshold technique represent outliers of individual LULC classes, but they are not necessarily associated with higher classification uncertainty. By contrast, unclassified pixels identified using the equal-likelihood technique are associated with higher classification uncertainty and they mostly occur on or near the borders of different land-cover.

Quantifying Uncertainty in Land-Use/Land-Cover Classification Accuracy: A Stochastic Simulation Approach

2021 ◽  
Vol 9 ◽  
Author(s):  
Ke-Sheng Cheng ◽  
Jia-Yi Ling ◽  
Teng-Wei Lin ◽  
Yin-Ting Liu ◽  
You-Chen Shen ◽  
...  
Keyword(s):  
Land Use ◽  
Land Cover ◽  
Stochastic Simulation ◽  
Classification Accuracy ◽  
Stochastic Simulations ◽  
Training Data ◽  
Land Use Land Cover ◽  
Simulation Approach ◽  
Data Set ◽  
Sample Data

In numerous applications of land-use/land-cover (LULC) classification, the classification rules are determined using a set of training data; thus, the results are inherently affected by uncertainty in the selection of those data. Few studies have assessed the accuracy of LULC classification with this consideration. In this article, we provide a general expression of various measures of classification accuracy with regard to the sample data set for classifier training and the sample data set for the evaluation of the classification results. We conducted stochastic simulations for LULC classification of a two-feature two-class case and a three-feature four-class case to show the uncertainties in the training sample and reference sample confusion matrices. A bootstrap simulation approach for establishing the 95% confidence interval of the classifier global accuracy was proposed and validated through rigorous stochastic simulation. Moreover, theoretical relationships among the producer accuracy, user accuracy, and overall accuracy were derived. The results demonstrate that the sample size of class-specific training data and the a priori probabilities of individual LULC classes must be jointly considered to ensure the correct determination of LULC classification accuracy.

scholarly journals Data Analysis With Shapley Values For Automatic Subject Selection in Alzheimer's Disease Data Sets Using Interpretable Machine Learning

2021 ◽  
Author(s):  
Louise Bloch ◽  
Christoph M. Friedrich
Keyword(s):  
Alzheimer’S Disease ◽  
Alzheimer's Disease ◽  
Test Data ◽  
Noisy Data ◽  
Training Data ◽  
Data Sets ◽  
Data Set ◽  
Model Interpretation ◽  
Percentage Points ◽  
Shapley Values

Abstract Background: The prediction of whether Mild Cognitive Impaired (MCI) subjects will prospectively develop Alzheimer's Disease (AD) is important for the recruitment and monitoring of subjects for therapy studies. Machine Learning (ML) is suitable to improve early AD prediction. The etiology of AD is heterogeneous, which leads to noisy data sets. Additional noise is introduced by multicentric study designs and varying acquisition protocols. This article examines whether an automatic and fair data valuation method based on Shapley values can identify subjects with noisy data. Methods: An ML-workow was developed and trained for a subset of the Alzheimer's Disease Neuroimaging Initiative (ADNI) cohort. The validation was executed for an independent ADNI test data set and for the Australian Imaging, Biomarker and Lifestyle Flagship Study of Ageing (AIBL) cohort. The workow included volumetric Magnetic Resonance Imaging (MRI) feature extraction, subject sample selection using data Shapley, Random Forest (RF) and eXtreme Gradient Boosting (XGBoost) for model training and Kernel SHapley Additive exPlanations (SHAP) values for model interpretation. This model interpretation enables clinically relevant explanation of individual predictions. Results: The XGBoost models which excluded 116 of the 467 subjects from the training data set based on their Logistic Regression (LR) data Shapley values outperformed the models which were trained on the entire training data set and which reached a mean classification accuracy of 58.54 % by 14.13 % (8.27 percentage points) on the independent ADNI test data set. The XGBoost models, which were trained on the entire training data set reached a mean accuracy of 60.35 % for the AIBL data set. An improvement of 24.86 % (15.00 percentage points) could be reached for the XGBoost models if those 72 subjects with the smallest RF data Shapley values were excluded from the training data set. Conclusion: The data Shapley method was able to improve the classification accuracies for the test data sets. Noisy data was associated with the number of ApoEϵ4 alleles and volumetric MRI measurements. Kernel SHAP showed that the black-box models learned biologically plausible associations.

An Incremental Isomap Method for Hyperspectral Dimensionality Reduction and Classification

2021 ◽  
Vol 87 (6) ◽  
pp. 445-455
Author(s):  
Yi Ma ◽  
Zezhong Zheng ◽  
Yutang Ma ◽  
Mingcang Zhu ◽  
Ran Huang ◽  
...  
Keyword(s):  
Manifold Learning ◽  
Nearest Neighbor ◽  
Hyperspectral Image ◽  
Hyperspectral Data ◽  
Training Data ◽  
Support Vector ◽  
Data Sets ◽  
K Nearest Neighbor ◽  
Data Set ◽  
Data Points

Many manifold learning algorithms conduct an eigen vector analysis on a data-similarity matrix with a size of N×N, where N is the number of data points. Thus, the memory complexity of the analysis is no less than O(N2). We pres- ent in this article an incremental manifold learning approach to handle large hyperspectral data sets for land use identification. In our method, the number of dimensions for the high-dimensional hyperspectral-image data set is obtained with the training data set. A local curvature varia- tion algorithm is utilized to sample a subset of data points as landmarks. Then a manifold skeleton is identified based on the landmarks. Our method is validated on three AVIRIS hyperspectral data sets, outperforming the comparison algorithms with a k–nearest-neighbor classifier and achieving the second best performance with support vector machine.

Simple Convolutional-Based Models: Are They Learning the Task or the Data?

2021 ◽  
pp. 1-17
Author(s):  
Luis Sa-Couto ◽  
Andreas Wichert
Keyword(s):  
Neural Networks ◽  
Pattern Recognition ◽  
Training Data ◽  
Model Complexity ◽  
Data Sets ◽  
Simple Task ◽  
Data Set ◽  
Knowing That ◽  
Handwritten Digit ◽  
End To End

Abstract Convolutional neural networks (CNNs) evolved from Fukushima's neocognitron model, which is based on the ideas of Hubel and Wiesel about the early stages of the visual cortex. Unlike other branches of neocognitron-based models, the typical CNN is based on end-to-end supervised learning by backpropagation and removes the focus from built-in invariance mechanisms, using pooling not as a way to tolerate small shifts but as a regularization tool that decreases model complexity. These properties of end-to-end supervision and flexibility of structure allow the typical CNN to become highly tuned to the training data, leading to extremely high accuracies on typical visual pattern recognition data sets. However, in this work, we hypothesize that there is a flip side to this capability, a hidden overfitting. More concretely, a supervised, backpropagation based CNN will outperform a neocognitron/map transformation cascade (MTCCXC) when trained and tested inside the same data set. Yet if we take both models trained and test them on the same task but on another data set (without retraining), the overfitting appears. Other neocognitron descendants like the What-Where model go in a different direction. In these models, learning remains unsupervised, but more structure is added to capture invariance to typical changes. Knowing that, we further hypothesize that if we repeat the same experiments with this model, the lack of supervision may make it worse than the typical CNN inside the same data set, but the added structure will make it generalize even better to another one. To put our hypothesis to the test, we choose the simple task of handwritten digit classification and take two well-known data sets of it: MNIST and ETL-1. To try to make the two data sets as similar as possible, we experiment with several types of preprocessing. However, regardless of the type in question, the results align exactly with expectation.

scholarly journals Geo-Object-Based Land Cover Map Update for High-Spatial-Resolution Remote Sensing Images via Change Detection and Label Transfer

2020 ◽  
Vol 12 (1) ◽  
pp. 174
Author(s):  
Tianjun Wu ◽  
Jiancheng Luo ◽  
Ya’nan Zhou ◽  
Changpeng Wang ◽  
Jiangbo Xi ◽  
...  
Keyword(s):  
Remote Sensing ◽  
Land Cover ◽  
Change Detection ◽  
Spatial Resolution ◽  
High Spatial Resolution ◽  
Data Set ◽  
Label Transfer ◽  
Object Based ◽  
Training Samples ◽  
Map Updating

Land cover (LC) information plays an important role in different geoscience applications such as land resources and ecological environment monitoring. Enhancing the automation degree of LC classification and updating at a fine scale by remote sensing has become a key problem, as the capability of remote sensing data acquisition is constantly being improved in terms of spatial and temporal resolution. However, the present methods of generating LC information are relatively inefficient, in terms of manually selecting training samples among multitemporal observations, which is becoming the bottleneck of application-oriented LC mapping. Thus, the objectives of this study are to speed up the efficiency of LC information acquisition and update. This study proposes a rapid LC map updating approach at a geo-object scale for high-spatial-resolution (HSR) remote sensing. The challenge is to develop methodologies for quickly sampling. Hence, the core step of our proposed methodology is an automatic method of collecting samples from historical LC maps through combining change detection and label transfer. A data set with Chinese Gaofen-2 (GF-2) HSR satellite images is utilized to evaluate the effectiveness of our method for multitemporal updating of LC maps. Prior labels in a historical LC map are certified to be effective in a LC updating task, which contributes to improve the effectiveness of the LC map update by automatically generating a number of training samples for supervised classification. The experimental outcomes demonstrate that the proposed method enhances the automation degree of LC map updating and allows for geo-object-based up-to-date LC mapping with high accuracy. The results indicate that the proposed method boosts the ability of automatic update of LC map, and greatly reduces the complexity of visual sample acquisition. Furthermore, the accuracy of LC type and the fineness of polygon boundaries in the updated LC maps effectively reflect the characteristics of geo-object changes on the ground surface, which makes the proposed method suitable for many applications requiring refined LC maps.

scholarly journals Training Data Selection for Annual Land Cover Classification for the Land Change Monitoring, Assessment, and Projection (LCMAP) Initiative

2020 ◽  
Vol 12 (4) ◽  
pp. 699 ◽  
Author(s):  
Qiang Zhou ◽  
Heather Tollerud ◽  
Christopher Barber ◽  
Kelcy Smith ◽  
Daniel Zelenak
Keyword(s):  
Land Cover ◽  
Land Surface ◽  
Classification Accuracy ◽  
Training Data ◽  
Land Change ◽  
Auxiliary Data ◽  
Surface Change ◽  
Land Cover Map ◽  
The U.S ◽  
Change Monitoring

The U.S. Geological Survey’s Land Change Monitoring, Assessment, and Projection (LCMAP) initiative involves detecting changes in land cover, use, and condition with the goal of producing land change information to improve the understanding of the Earth system and provide insights on the impacts of land surface change on society. The change detection method ingests all available high-quality data from the Landsat archive in a time series approach to identify the timing and location of land surface change. Annual thematic land cover maps are then produced by classifying time series models. In this paper, we describe the optimization of the classification method used to derive the thematic land cover product. We investigated the influences of auxiliary data, sample size, and training from different sources such as the U.S. Geological Survey’s Land Cover Trends project and National Land Cover Database (NLCD 2001 and NLCD 2011). The results were evaluated and validated based on independent data from the training dataset. We found that refining the auxiliary data effectively reduced artifacts in the thematic land cover map that are related to data availability. We improved the classification accuracy and stability considerably by using a total of 20 million training pixels with a minimum of 600,000 and a maximum of 8 million training pixels per class within geographic windows consisting of nine Analysis Ready Data tiles (450 by 450 km2). Comparisons revealed that the NLCD 2001 training data delivered the best classification accuracy. Compared to the original LCMAP classification strategy used for early evaluation (e.g., Trends training data, 20,000 samples), the optimized classification strategy improved the annual land cover map accuracy by an average of 10%.

scholarly journals Deep Neural Network Algorithm Feedback Model with Behavioral Intelligence and Forecast Accuracy

Symmetry ◽  
2020 ◽  
Vol 12 (9) ◽  
pp. 1465
Author(s):  
Taikyeong Jeong
Keyword(s):  
Neural Network ◽  
Classification Accuracy ◽  
Deep Neural Network ◽  
Time Series Data ◽  
Training Data ◽  
Series Data ◽  
Context Aware ◽  
Data Set ◽  
Feedback Model ◽  
Data Connection

When attempting to apply a large-scale database that holds the behavioral intelligence training data of deep neural networks, the classification accuracy of the artificial intelligence algorithm needs to reflect the behavioral characteristics of the individual. When a change in behavior is recognized, that is, a feedback model based on a data connection model is applied, an analysis of time series data is performed by extracting feature vectors and interpolating data in a deep neural network to overcome the limitations of the existing statistical analysis. Using the results of the first feedback model as inputs to the deep neural network and, furthermore, as the input values of the second feedback model, and interpolating the behavioral intelligence data, that is, context awareness and lifelog data, including physical activities, involves applying the most appropriate conditions. The results of this study show that this method effectively improves the accuracy of the artificial intelligence results. In this paper, through an experiment, after extracting the feature vector of a deep neural network and restoring the missing value, the classification accuracy was verified to improve by about 20% on average. At the same time, by adding behavioral intelligence data to the time series data, a new data connection model, the Deep Neural Network Feedback Model, was proposed, and it was verified that the classification accuracy can be improved by about 8 to 9% on average. Based on the hypothesis, the F (X′) = X model was applied to thoroughly classify the training data set and test data set to present a symmetrical balance between the data connection model and the context-aware data. In addition, behavioral activity data were extrapolated in terms of context-aware and forecasting perspectives to prove the results of the experiment.

scholarly journals An assessment of Bayesian bias estimator for numerical weather prediction

2008 ◽  
Vol 15 (6) ◽  
pp. 1013-1022 ◽  
Author(s):  
J. Son ◽  
D. Hou ◽  
Z. Toth
Keyword(s):  
Time Series ◽  
Numerical Weather Prediction ◽  
Sampling Error ◽  
Weather Prediction ◽  
Training Data ◽  
Statistical Characteristics ◽  
Forecast Errors ◽  
Data Sets ◽  
Data Set ◽  
Numerical Weather

Abstract. Various statistical methods are used to process operational Numerical Weather Prediction (NWP) products with the aim of reducing forecast errors and they often require sufficiently large training data sets. Generating such a hindcast data set for this purpose can be costly and a well designed algorithm should be able to reduce the required size of these data sets. This issue is investigated with the relatively simple case of bias correction, by comparing a Bayesian algorithm of bias estimation with the conventionally used empirical method. As available forecast data sets are not large enough for a comprehensive test, synthetically generated time series representing the analysis (truth) and forecast are used to increase the sample size. Since these synthetic time series retained the statistical characteristics of the observations and operational NWP model output, the results of this study can be extended to real observation and forecasts and this is confirmed by a preliminary test with real data. By using the climatological mean and standard deviation of the meteorological variable in consideration and the statistical relationship between the forecast and the analysis, the Bayesian bias estimator outperforms the empirical approach in terms of the accuracy of the estimated bias, and it can reduce the required size of the training sample by a factor of 3. This advantage of the Bayesian approach is due to the fact that it is less liable to the sampling error in consecutive sampling. These results suggest that a carefully designed statistical procedure may reduce the need for the costly generation of large hindcast datasets.

SaltSeg: Automatic 3D salt segmentation using a deep convolutional neural network

2019 ◽  
Vol 7 (3) ◽  
pp. SE113-SE122 ◽  
Author(s):  
Yunzhi Shi ◽  
Xinming Wu ◽  
Sergey Fomel
Keyword(s):  
Large Scale ◽  
Model Building ◽  
Ground Truth ◽  
Velocity Model ◽  
Training Data ◽  
Data Sets ◽  
Validation Data ◽  
Data Set ◽  
Seismic Image ◽  
Data Generator

Salt boundary interpretation is important for the understanding of salt tectonics and velocity model building for seismic migration. Conventional methods consist of computing salt attributes and extracting salt boundaries. We have formulated the problem as 3D image segmentation and evaluated an efficient approach based on deep convolutional neural networks (CNNs) with an encoder-decoder architecture. To train the model, we design a data generator that extracts randomly positioned subvolumes from large-scale 3D training data set followed by data augmentation, then feed a large number of subvolumes into the network while using salt/nonsalt binary labels generated by thresholding the velocity model as ground truth labels. We test the model on validation data sets and compare the blind test predictions with the ground truth. Our results indicate that our method is capable of automatically capturing subtle salt features from the 3D seismic image with less or no need for manual input. We further test the model on a field example to indicate the generalization of this deep CNN method across different data sets.

scholarly journals PUB-SalNet: A Pre-Trained Unsupervised Self-Aware Backpropagation Network for Biomedical Salient Segmentation

Algorithms ◽  
2020 ◽  
Vol 13 (5) ◽  
pp. 126 ◽  
Author(s):  
Feiyang Chen ◽  
Ying Jiang ◽  
Xiangrui Zeng ◽  
Jing Zhang ◽  
Xin Gao ◽  
...  
Keyword(s):  
Training Data ◽  
Data Sets ◽  
Local Similarity ◽  
Biomedical Data ◽  
Data Set ◽  
Biomedical Image ◽  
Feature Representations ◽  
3D Data ◽  
Supervised Methods ◽  
Data Driven Decisions

Salient segmentation is a critical step in biomedical image analysis, aiming to cut out regions that are most interesting to humans. Recently, supervised methods have achieved promising results in biomedical areas, but they depend on annotated training data sets, which requires labor and proficiency in related background knowledge. In contrast, unsupervised learning makes data-driven decisions by obtaining insights directly from the data themselves. In this paper, we propose a completely unsupervised self-aware network based on pre-training and attentional backpropagation for biomedical salient segmentation, named as PUB-SalNet. Firstly, we aggregate a new biomedical data set from several simulated Cellular Electron Cryo-Tomography (CECT) data sets featuring rich salient objects, different SNR settings, and various resolutions, which is called SalSeg-CECT. Based on the SalSeg-CECT data set, we then pre-train a model specially designed for biomedical tasks as a backbone module to initialize network parameters. Next, we present a U-SalNet network to learn to selectively attend to salient objects. It includes two types of attention modules to facilitate learning saliency through global contrast and local similarity. Lastly, we jointly refine the salient regions together with feature representations from U-SalNet, with the parameters updated by self-aware attentional backpropagation. We apply PUB-SalNet for analysis of 2D simulated and real images and achieve state-of-the-art performance on simulated biomedical data sets. Furthermore, our proposed PUB-SalNet can be easily extended to 3D images. The experimental results on the 2d and 3d data sets also demonstrate the generalization ability and robustness of our method.

Sign in / Sign up

Export Citation Format

Share Document