scholarly journals Probing the Effect of Selection Bias on Generalization: A Thought Experiment

2021 ◽  
Author(s):  
John Tsotsos ◽  
Jun Luo
Keyword(s):  
Data Collection ◽  
Selection Bias ◽  
Visual Recognition ◽  
Thought Experiment ◽  
System Development ◽  
Training Data ◽  
Real Problem ◽  
Data Sets ◽  
Training Set ◽  
Unseen Data

Abstract Learned systems in the domain of visual recognition and cognition impress in part because even though they are trained with datasets many orders of magnitude smaller than the full population of possible images, they exhibit sufficient generalization to be applicable to new and previously unseen data. Since training data sets typically represent such a small sampling of any domain, the possibility of bias in their composition is very real. But what are the limits of generalization given such bias, and up to what point might it be sufficient for a real problem task? Although many have examined issues regarding generalization from several perspectives, this question may require examining the data itself. Here, we focus on the characteristics of the training data that may play a role. Other disciplines have grappled with these problems also, most interestingly epidemiology, where experimental bias is a critical concern. The range and nature of data biases seen clinically are really quite relatable to learned vision systems. One obvious way to deal with bias is to ensure a large enough training set, but this might be infeasible for many domains. Another approach might be to perform a statistical analysis of the actual training set, to determine if all aspects of the domain are fairly captured. This too is difficult, in part because the full set of important variables might not be known, or perhaps not even knowable. Here, we try a different, simpler, approach in the tradition of the Thought Experiment, whose most famous instance is perhaps Schrödinger's Cat, to address part of these problems. There are many types of bias as will be seen, but we focus only on one, selection bias. The point of the thought experiment is not to demonstrate problems with all learned systems. Rather, this might be a simple theoretical tool to probe into bias during data collection to highlight deficiencies that might then deserve extra attention either in data collection or system development.

scholarly journals CHIRPS: Explaining random forest classification

2020 ◽  
Vol 53 (8) ◽  
pp. 5747-5788
Author(s):  
Julian Hatwell ◽  
Mohamed Medhat Gaber ◽  
R. Muhammad Atif Azad
Keyword(s):  
Random Forest ◽  
Pattern Mining ◽  
Frequent Pattern Mining ◽  
Training Data ◽  
Frequent Pattern ◽  
Data Sets ◽  
Random Forest Classification ◽  
Human In The Loop ◽  
Forest Classification ◽  
Unseen Data

Abstract Modern machine learning methods typically produce “black box” models that are opaque to interpretation. Yet, their demand has been increasing in the Human-in-the-Loop processes, that is, those processes that require a human agent to verify, approve or reason about the automated decisions before they can be applied. To facilitate this interpretation, we propose Collection of High Importance Random Path Snippets (CHIRPS); a novel algorithm for explaining random forest classification per data instance. CHIRPS extracts a decision path from each tree in the forest that contributes to the majority classification, and then uses frequent pattern mining to identify the most commonly occurring split conditions. Then a simple, conjunctive form rule is constructed where the antecedent terms are derived from the attributes that had the most influence on the classification. This rule is returned alongside estimates of the rule’s precision and coverage on the training data along with counter-factual details. An experimental study involving nine data sets shows that classification rules returned by CHIRPS have a precision at least as high as the state of the art when evaluated on unseen data (0.91–0.99) and offer a much greater coverage (0.04–0.54). Furthermore, CHIRPS uniquely controls against under- and over-fitting solutions by maximising novel objective functions that are better suited to the local (per instance) explanation setting.

scholarly journals Robust Federated Learning via Collaborative Machine Teaching

2020 ◽  
Vol 34 (04) ◽  
pp. 4075-4082
Author(s):  
Yufei Han ◽  
Xiangliang Zhang
Keyword(s):  
Teaching And Learning ◽  
Service Providers ◽  
Teaching Method ◽  
Training Data ◽  
Data Sets ◽  
Quality Of Data ◽  
Training Set ◽  
In The Wild ◽  
Model Training ◽  
Set Up

For federated learning systems deployed in the wild, data flaws hosted on local agents are widely witnessed. On one hand, given a large amount (e.g. over 60%) of training data are corrupted by systematic sensor noise and environmental perturbations, the performances of federated model training can be degraded significantly. On the other hand, it is prohibitively expensive for either clients or service providers to set up manual sanitary checks to verify the quality of data instances. In our study, we echo this challenge by proposing a collaborative and privacy-preserving machine teaching method. Specifically, we use a few trusted instances provided by teachers as benign examples in the teaching process. Our collaborative teaching approach seeks jointly the optimal tuning on the distributed training set, such that the model learned from the tuned training set predicts labels of the trusted items correctly. The proposed method couples the process of teaching and learning and thus produces directly a robust prediction model despite the extremely pervasive systematic data corruption. The experimental study on real benchmark data sets demonstrates the validity of our method.

A method for adequate selection of training data sets to reconstruct seismic field data using a convolutional U-Net

Geophysics ◽  
2021 ◽  
pp. 1-103
Author(s):  
Jiho Park ◽  
Jihun Choi ◽  
Soon Jee Seol ◽  
Joongmoo Byun ◽  
Young Kim
Keyword(s):  
Data Analysis ◽  
Seismic Data ◽  
Field Data ◽  
Training Data ◽  
Data Reconstruction ◽  
Data Sets ◽  
Training Set ◽  
Data Set ◽  
Seismic Data Reconstruction ◽  
Target Data

Deep learning (DL) methods are recently introduced for seismic signal processing. Using DL methods, many researchers have adopted these novel techniques in an attempt to construct a DL model for seismic data reconstruction. The performance of DL-based methods depends heavily on what is learned from the training data. We focus on constructing the DL model that well reflect the features of target data sets. The main goal is to integrate DL with an intuitive data analysis approach that compares similar patterns prior to the DL training stage. We have developed a two-sequential method consisting of two stage: (i) analyzing training and target data sets simultaneously for determining target-informed training set and (ii) training the DL model with this training data set to effectively interpolate the seismic data. Here, we introduce the convolutional autoencoder t-distributed stochastic neighbor embedding (CAE t-SNE) analysis that can provide the insight into the results of interpolation through the analysis of both the training and target data sets prior to DL model training. The proposed method were tested with synthetic and field data. Dense seismic gathers (e.g. common-shot gathers; CSGs) were used as a labeled training data set, and relatively sparse seismic gather (e.g. common-receiver gathers; CRGs) were reconstructed in both cases. The reconstructed results and SNRs demonstrated that the training data can be efficiently selected using CAE t-SNE analysis and the spatial aliasing of CRGs was successfully alleviated by the trained DL model with this training data, which contain target features. These results imply that the data analysis for selecting target-informed training set is very important for successful DL interpolation. Additionally, the proposed analysis method can also be applied to investigate the similarities between training and target data sets for another DL-based seismic data reconstruction tasks.

Work Site Trip Reduction Model and Manual

2005 ◽  
Vol 1924 (1) ◽  
pp. 197-206
Author(s):  
Philip L. Winters ◽  
Rafael A. Perez ◽  
Ajay D. Joshi ◽  
Jennifer Perone
Keyword(s):  
Los Angeles ◽  
Urban Areas ◽  
Model Building ◽  
The United States ◽  
Training Data ◽  
Data Sets ◽  
Statistical Regression ◽  
Reduction Model ◽  
Work Site ◽  
Unseen Data

Today's transportation professionals often use the ITE Trip Generation Manual and the Parking Generation Manual for estimating future traffic volumes to base off-site transportation improvements and identify parking requirements. But these manuals are inadequate for assessing the claims made by specific transportation demand management (TDM) programs in reducing vehicle trips by a certain amount at particular work sites. This paper presents a work site trip reduction model (WTRM) that can help transportation professionals in assessing those claims. WTRM was built on data from three urban areas in the United States: Los Angeles, California; Tucson, Arizona; and nine counties in Washington State. The data consist of work sites’ employee modal characteristics aggregated at the employer level and a listing of incentives and amenities offered by employers. The dependent variable chosen was the change in vehicle trip rate that correlated with the goals of TDM programs. Two different approaches were used in the model-building process: linear statistical regression and nonlinear neural networks. For performance evaluation the data sets were divided into two disjoint sets: a training set, which was used to build the models, and a validation set, which was used as unseen data to evaluate the models. Because the number of data samples varied from the three areas, two training data sets were formed: one consisted of all training data samples from three areas and the other contained equally sampled training data from the three areas. The best model was the neural net model built on equally sampled training data.

scholarly journals Learning When Training Data are Costly: The Effect of Class Distribution on Tree Induction

2003 ◽  
Vol 19 ◽  
pp. 315-354 ◽  
Author(s):  
G. M. Weiss ◽  
F. Provost
Keyword(s):  
Inductive Learning ◽  
Practical Importance ◽  
Classification Performance ◽  
Training Data ◽  
Data Sets ◽  
Training Set ◽  
Class Distribution ◽  
Classifier Performance ◽  
Balanced Distribution ◽  
Training Examples

For large, real-world inductive learning problems, the number of training examples often must be limited due to the costs associated with procuring, preparing, and storing the training examples and/or the computational costs associated with learning from them. In such circumstances, one question of practical importance is: if only n training examples can be selected, in what proportion should the classes be represented? In this article we help to answer this question by analyzing, for a fixed training-set size, the relationship between the class distribution of the training data and the performance of classification trees induced from these data. We study twenty-six data sets and, for each, determine the best class distribution for learning. The naturally occurring class distribution is shown to generally perform well when classifier performance is evaluated using undifferentiated error rate (0/1 loss). However, when the area under the ROC curve is used to evaluate classifier performance, a balanced distribution is shown to perform well. Since neither of these choices for class distribution always generates the best-performing classifier, we introduce a budget-sensitive progressive sampling algorithm for selecting training examples based on the class associated with each example. An empirical analysis of this algorithm shows that the class distribution of the resulting training set yields classifiers with good (nearly-optimal) classification performance.

scholarly journals The Effect of Clustering with a minimum Pattern of Features Extraction for Person Recognition

2021 ◽  
Vol 14 (2) ◽  
pp. 120-128
Author(s):  
Mohammed Ehsan Safi ◽  
Eyad I. Abbas
Keyword(s):  
Recognition Rate ◽  
Principal Component ◽  
Training Data ◽  
Data Sets ◽  
Training Set ◽  
Data Set ◽  
Person Recognition ◽  
Personal Recognition ◽  
Application Requirements ◽  
Training Sets

In personal image recognition algorithms, two effective factors govern the system’s evaluation, recognition rate and size of the database. Unfortunately, the recognition rate proportional to the increase in training sets. Consequently, that increases the processing time and memory limitation problems. This paper’s main goal was to present a robust algorithm with minimum data sets and a high recognition rate. Images for ten persons were chosen as a database, nine images for each individual as the full version of the training data set, and one image for each person out of the training set as a test pattern before the database reduction procedure. The proposed algorithm integrates Principal Component Analysis (PCA) as a feature extraction technique with the minimum means of clusters and Euclidean Distance to achieve personal recognition. After indexing the training set for each person, the clustering of the differences is determined. The recognition of the person represented by the minimum mean index; this process returned with each reduction. The experimental results show that the recognition rate is 100% despite reducing the training sets to 44%, while the recognition rate decrease to 70% when the reduction reaches 89%. The clear picture out is the results of the proposed system support the idea of the redaction of training sets in addition to obtaining a high recognition rate based on application requirements.

scholarly journals Optimising training data for ANNs with Genetic Algorithms

2006 ◽  
Vol 3 (2) ◽  
pp. 285-297 ◽  
Author(s):  
R. G. Kamp ◽  
H. H. G. Savenije
Keyword(s):  
Genetic Algorithms ◽  
Hydrological Model ◽  
Training Data ◽  
Data Sets ◽  
Training Phase ◽  
Rainfall Runoff ◽  
Input Output ◽  
Training Set ◽  
Hydraulic Flow ◽  
Flow Modelling

Abstract. Artificial Neural Networks have proven to be good modelling tools in hydrology for rainfall-runoff modelling and hydraulic flow modelling. Representative data sets are necessary for the training phase in which the ANN learns the model's input-output relations. Good and representative training data is not always available. In this publication Genetic Algorithms are used to optimise training data sets. The approach is tested with an existing hydrological model in The Netherlands. The optimised training set resulted in significant better training data.

Participant recruitment and data collection through Facebook: the role of personality factors

2017 ◽  
Author(s):  
Sean Chandler Rife ◽  
Kelly L. Cate ◽  
Michal Kosinski ◽  
David Stillwell
Keyword(s):  
Data Collection ◽  
Social Research ◽  
Personality Factors ◽  
The Internet ◽  
Data Sets ◽  
Social Networking Service ◽  
Participant Recruitment ◽  
Conducting Research ◽  
Promising Avenue

As participant recruitment and data collection over the Internet have become more common, numerous observers have expressed concern regarding the validity of research conducted in this fashion. One growing method of conducting research over the Internet involves recruiting participants and administering questionnaires over Facebook, the world’s largest social networking service. If Facebook is to be considered a viable platform for social research, it is necessary to demonstrate that Facebook users are sufficiently heterogeneous and that research conducted through Facebook is likely to produce results that can be generalized to a larger population. The present study examines these questions by comparing demographic and personality data collected over Facebook with data collected through a standalone website, and data collected from college undergraduates at two universities. Results indicate that statistically significant differences exist between Facebook data and the comparison data-sets, but since 80% of analyses exhibited partial η2 < .05, such differences are small or practically nonsignificant in magnitude. We conclude that Facebook is a viable research platform, and that recruiting Facebook users for research purposes is a promising avenue that offers numerous advantages over traditional samples.

MULFE: Multi-Label Learning via Label-Specific Feature Space Ensemble

2021 ◽  
Vol 16 (1) ◽  
pp. 1-24
Author(s):  
Yaojin Lin ◽  
Qinghua Hu ◽  
Jinghua Liu ◽  
Xingquan Zhu ◽  
Xindong Wu
Keyword(s):  
Empirical Studies ◽  
Feature Space ◽  
Training Data ◽  
Data Sets ◽  
Learning Framework ◽  
Feature Spaces ◽  
Public Data ◽  
Margin Distribution ◽  
Label Correlations ◽  
Label Correlation

In multi-label learning, label correlations commonly exist in the data. Such correlation not only provides useful information, but also imposes significant challenges for multi-label learning. Recently, label-specific feature embedding has been proposed to explore label-specific features from the training data, and uses feature highly customized to the multi-label set for learning. While such feature embedding methods have demonstrated good performance, the creation of the feature embedding space is only based on a single label, without considering label correlations in the data. In this article, we propose to combine multiple label-specific feature spaces, using label correlation, for multi-label learning. The proposed algorithm, mu lti- l abel-specific f eature space e nsemble (MULFE), takes consideration label-specific features, label correlation, and weighted ensemble principle to form a learning framework. By conducting clustering analysis on each label’s negative and positive instances, MULFE first creates features customized to each label. After that, MULFE utilizes the label correlation to optimize the margin distribution of the base classifiers which are induced by the related label-specific feature spaces. By combining multiple label-specific features, label correlation based weighting, and ensemble learning, MULFE achieves maximum margin multi-label classification goal through the underlying optimization framework. Empirical studies on 10 public data sets manifest the effectiveness of MULFE.

scholarly journals Experiments of Image Classification Using Dissimilarity Spaces Built with Siamese Networks

Sensors ◽  
2021 ◽  
Vol 21 (5) ◽  
pp. 1573
Author(s):  
Loris Nanni ◽  
Giovanni Minchio ◽  
Sheryl Brahnam ◽  
Gianluca Maguolo ◽  
Alessandra Lumini
Keyword(s):  
Vector Space ◽  
Image Classification ◽  
Ad Hoc ◽  
Feature Space ◽  
Medical Data ◽  
Training Data ◽  
Data Sets ◽  
Large Set ◽  
Clustering Methods ◽  
Siamese Networks

Traditionally, classifiers are trained to predict patterns within a feature space. The image classification system presented here trains classifiers to predict patterns within a vector space by combining the dissimilarity spaces generated by a large set of Siamese Neural Networks (SNNs). A set of centroids from the patterns in the training data sets is calculated with supervised k-means clustering. The centroids are used to generate the dissimilarity space via the Siamese networks. The vector space descriptors are extracted by projecting patterns onto the similarity spaces, and SVMs classify an image by its dissimilarity vector. The versatility of the proposed approach in image classification is demonstrated by evaluating the system on different types of images across two domains: two medical data sets and two animal audio data sets with vocalizations represented as images (spectrograms). Results show that the proposed system’s performance competes competitively against the best-performing methods in the literature, obtaining state-of-the-art performance on one of the medical data sets, and does so without ad-hoc optimization of the clustering methods on the tested data sets.

Sign in / Sign up

Export Citation Format

Share Document