scholarly journals Adaptive Double-Exploration Tradeoff for Outlier Detection

2020 ◽  
Vol 34 (04) ◽  
pp. 6837-6844
Author(s):  
Xiaojin Zhang ◽  
Honglei Zhuang ◽  
Shengyu Zhang ◽  
Yuan Zhou
Keyword(s):  
Confidence Interval ◽  
Outlier Detection ◽  
Real World ◽  
Efficient Algorithm ◽  
Experimental Results ◽  
Sample Complexity ◽  
Bandit Problem ◽  
Real World Datasets ◽  
Synthetic Datasets ◽  
The Individual

We study a variant of the thresholding bandit problem (TBP) in the context of outlier detection, where the objective is to identify the outliers whose rewards are above a threshold. Distinct from the traditional TBP, the threshold is defined as a function of the rewards of all the arms, which is motivated by the criterion for identifying outliers. The learner needs to explore the rewards of the arms as well as the threshold. We refer to this problem as "double exploration for outlier detection". We construct an adaptively updated confidence interval for the threshold, based on the estimated value of the threshold in the previous rounds. Furthermore, by automatically trading off exploring the individual arms and exploring the outlier threshold, we provide an efficient algorithm in terms of the sample complexity. Experimental results on both synthetic datasets and real-world datasets demonstrate the efficiency of our algorithm.

scholarly journals OFCOD: On the Fly Clustering Based Outlier Detection Framework

Data ◽  
2020 ◽  
Vol 6 (1) ◽  
pp. 1
Author(s):  
Ahmed Elmogy ◽  
Hamada Rizk ◽  
Amany M. Sarhan
Keyword(s):  
Data Mining ◽  
Image Processing ◽  
Intrusion Detection ◽  
Real Time ◽  
Outlier Detection ◽  
Real World ◽  
Medical Data ◽  
Experimental Results ◽  
Real Time Applications ◽  
Real World Datasets

In data mining, outlier detection is a major challenge as it has an important role in many applications such as medical data, image processing, fraud detection, intrusion detection, and so forth. An extensive variety of clustering based approaches have been developed to detect outliers. However they are by nature time consuming which restrict their utilization with real-time applications. Furthermore, outlier detection requests are handled one at a time, which means that each request is initiated individually with a particular set of parameters. In this paper, the first clustering based outlier detection framework, (On the Fly Clustering Based Outlier Detection (OFCOD)) is presented. OFCOD enables analysts to effectively find out outliers on time with request even within huge datasets. The proposed framework has been tested and evaluated using two real world datasets with different features and applications; one with 699 records, and another with five millions records. The experimental results show that the performance of the proposed framework outperforms other existing approaches while considering several evaluation metrics.

scholarly journals Selective oversampling approach for strongly imbalanced data

2021 ◽  
Vol 7 ◽  
pp. e604
Author(s):  
Peter Gnip ◽  
Liberios Vokorokos ◽  
Peter Drotár
Keyword(s):  
Outlier Detection ◽  
Real World ◽  
State Of The Art ◽  
Imbalanced Data ◽  
Prediction Performance ◽  
Classifier Performance ◽  
Real World Applications ◽  
Real World Datasets ◽  
Synthetic Datasets ◽  
Representative Samples

Challenges posed by imbalanced data are encountered in many real-world applications. One of the possible approaches to improve the classifier performance on imbalanced data is oversampling. In this paper, we propose the new selective oversampling approach (SOA) that first isolates the most representative samples from minority classes by using an outlier detection technique and then utilizes these samples for synthetic oversampling. We show that the proposed approach improves the performance of two state-of-the-art oversampling methods, namely, the synthetic minority oversampling technique and adaptive synthetic sampling. The prediction performance is evaluated on four synthetic datasets and four real-world datasets, and the proposed SOA methods always achieved the same or better performance than other considered existing oversampling methods.

scholarly journals Self-weighted Multiview Clustering with Multiple Graphs

2017 ◽  
Author(s):  
Feiping Nie ◽  
Jing Li ◽  
Xuelong Li
Keyword(s):  
Real World ◽  
Spectral Clustering ◽  
Experimental Results ◽  
Clustering Method ◽  
Elegant Method ◽  
Multiview Learning ◽  
Cluster Label ◽  
Real World Datasets ◽  
Synthetic Datasets ◽  
Multiview Clustering

In multiview learning, it is essential to assign a reasonable weight to each view according to its importance. Thus, for multiview clustering task, a wise and elegant method should achieve clustering multiview data while learning the view weights. In this paper, we address this problem by exploring a Laplacian rank constrained graph, which can be approximately as the centroid of the built graph for each view with different confidences. We start our work with a natural thought that the weights can be learned by introducing a hyperparameter. By analyzing the weakness of it, we further propose a new multiview clustering method which is totally self-weighted. Furthermore, once the target graph is obtained in our models, we can directly assign the cluster label to each data point and do not need any postprocessing such as $K$-means in standard spectral clustering. Evaluations on two synthetic datasets prove the effectiveness of our methods. Compared with several representative graph-based multiview clustering approaches on four real-world datasets, experimental results demonstrate that the proposed methods achieve the better performances and our new clustering method is more practical to use.

Review Summary Generation in Online Systems: Frameworks for Supervised and Unsupervised Scenarios

2021 ◽  
Vol 15 (3) ◽  
pp. 1-33
Author(s):  
Wenjun Jiang ◽  
Jing Chen ◽  
Xiaofei Ding ◽  
Jie Wu ◽  
Jiawei He ◽  
...  
Keyword(s):  
Decision Making ◽  
Real World ◽  
Text Summarization ◽  
Experimental Results ◽  
Product Review ◽  
Comprehensive Review ◽  
Online Systems ◽  
Real World Datasets ◽  
Different Characteristics

In online systems, including e-commerce platforms, many users resort to the reviews or comments generated by previous consumers for decision making, while their time is limited to deal with many reviews. Therefore, a review summary, which contains all important features in user-generated reviews, is expected. In this article, we study “how to generate a comprehensive review summary from a large number of user-generated reviews.” This can be implemented by text summarization, which mainly has two types of extractive and abstractive approaches. Both of these approaches can deal with both supervised and unsupervised scenarios, but the former may generate redundant and incoherent summaries, while the latter can avoid redundancy but usually can only deal with short sequences. Moreover, both approaches may neglect the sentiment information. To address the above issues, we propose comprehensive Review Summary Generation frameworks to deal with the supervised and unsupervised scenarios. We design two different preprocess models of re-ranking and selecting to identify the important sentences while keeping users’ sentiment in the original reviews. These sentences can be further used to generate review summaries with text summarization methods. Experimental results in seven real-world datasets (Idebate, Rotten Tomatoes Amazon, Yelp, and three unlabelled product review datasets in Amazon) demonstrate that our work performs well in review summary generation. Moreover, the re-ranking and selecting models show different characteristics.

scholarly journals Multiple Noisy Label Distribution Propagation for Crowdsourcing

2019 ◽  
Author(s):  
Hao Zhang ◽  
Liangxiao Jiang ◽  
Wenqiang Xu
Keyword(s):  
Supervised Learning ◽  
Real World ◽  
Effective Means ◽  
Ground Truth ◽  
Cost Effective ◽  
Nearest Neighbors ◽  
True Label ◽  
Real World Datasets ◽  
The Individual ◽  
Label Distribution

Crowdsourcing services provide a fast, efficient, and cost-effective means of obtaining large labeled data for supervised learning. Ground truth inference, also called label integration, designs proper aggregation strategies to infer the unknown true label of each instance from the multiple noisy label set provided by ordinary crowd workers. However, to the best of our knowledge, nearly all existing label integration methods focus solely on the multiple noisy label set itself of the individual instance while totally ignoring the intercorrelation among multiple noisy label sets of different instances. To solve this problem, a multiple noisy label distribution propagation (MNLDP) method is proposed in this study. MNLDP first transforms the multiple noisy label set of each instance into its multiple noisy label distribution and then propagates its multiple noisy label distribution to its nearest neighbors. Consequently, each instance absorbs a fraction of the multiple noisy label distributions from its nearest neighbors and yet simultaneously maintains a fraction of its own original multiple noisy label distribution. Promising experimental results on simulated and real-world datasets validate the effectiveness of our proposed method.

An Efficient Distance and Density Based Outlier Detection Approach

2012 ◽  
Vol 155-156 ◽  
pp. 342-347 ◽  
Author(s):  
Xun Biao Zhong ◽  
Xiao Xia Huang
Keyword(s):  
Outlier Detection ◽  
Real World ◽  
High Performance ◽  
Detection Problem ◽  
Empirical Results ◽  
Detection Approach ◽  
Real World Datasets ◽  
Good Detection

In order to solve the density based outlier detection problem with low accuracy and high computation, a variance of distance and density (VDD) measure is proposed in this paper. And the k-means clustering and score based VDD (KSVDD) approach proposed can efficiently detect outliers with high performance. For illustration, two real-world datasets are utilized to show the feasibility of the approach. Empirical results show that KSVDD has a good detection precision.

SOCIAL INTEREST FOR USER SELECTING ITEMS IN RECOMMENDER SYSTEMS

2013 ◽  
Vol 24 (04) ◽  
pp. 1350022 ◽  
Author(s):  
DA-CHENG NIE ◽  
MING-JING DING ◽  
YAN FU ◽  
JUN-LIN ZHOU ◽  
ZI-KE ZHANG
Keyword(s):  
Recommender Systems ◽  
Real World ◽  
Social Interest ◽  
Experimental Results ◽  
Simple Method ◽  
The Social ◽  
Social Interests ◽  
Similarity Computation ◽  
Real World Datasets

Recommender systems have developed rapidly and successfully. The system aims to help users find relevant items from a potentially overwhelming set of choices. However, most of the existing recommender algorithms focused on the traditional user-item similarity computation, other than incorporating the social interests into the recommender systems. As we know, each user has their own preference field, they may influence their friends' preference in their expert field when considering the social interest on their friends' item collecting. In order to model this social interest, in this paper, we proposed a simple method to compute users' social interest on the specific items in the recommender systems, and then integrate this social interest with similarity preference. The experimental results on two real-world datasets Epinions and Friendfeed show that this method can significantly improve not only the algorithmic precision-accuracy but also the diversity-accuracy.

scholarly journals FairLOF: Fairness in Outlier Detection

2021 ◽  
Author(s):  
Deepak P ◽  
Savitha Sam Abraham
Keyword(s):  
Marital Status ◽  
Outlier Detection ◽  
Real World ◽  
Detection Method ◽  
Empirical Evaluation ◽  
Evaluation Framework ◽  
Real World Datasets

AbstractAn outlier detection method may be considered fair over specified sensitive attributes if the results of outlier detection are not skewed toward particular groups defined on such sensitive attributes. In this paper, we consider the task of fair outlier detection. Our focus is on the task of fair outlier detection over multiple multi-valued sensitive attributes (e.g., gender, race, religion, nationality and marital status, among others), one that has broad applications across modern data scenarios. We propose a fair outlier detection method, FairLOF, that is inspired by the popular LOF formulation for neighborhood-based outlier detection. We outline ways in which unfairness could be induced within LOF and develop three heuristic principles to enhance fairness, which form the basis of the FairLOF method. Being a novel task, we develop an evaluation framework for fair outlier detection, and use that to benchmark FairLOF on quality and fairness of results. Through an extensive empirical evaluation over real-world datasets, we illustrate that FairLOF is able to achieve significant improvements in fairness at sometimes marginal degradations on result quality as measured against the fairness-agnostic LOF method. We also show that a generalization of our method, named FairLOF-Flex, is able to open possibilities of further deepening fairness in outlier detection beyond what is offered by FairLOF.

scholarly journals An Improved Semisupervised Outlier Detection Algorithm Based on Adaptive Feature Weighted Clustering

2016 ◽  
Vol 2016 ◽  
pp. 1-14 ◽  
Author(s):  
Tingquan Deng ◽  
Jinhong Yang
Keyword(s):  
Outlier Detection ◽  
Real World ◽  
Detection Algorithm ◽  
Membership Degree ◽  
Weighted Clustering ◽  
Detection Strategy ◽  
Series Of Experiments ◽  
Effectiveness And Efficiency ◽  
Real World Datasets ◽  
Normal Object

There exist already various approaches to outlier detection, in which semisupervised methods achieve encouraging superiority due to the introduction of prior knowledge. In this paper, an adaptive feature weighted clustering-based semisupervised outlier detection strategy is proposed. This method maximizes the membership degree of a labeled normal object to the cluster it belongs to and minimizes the membership degrees of a labeled outlier to all clusters. In consideration of distinct significance of features or components in a dataset in determining an object being an inlier or outlier, each feature is adaptively assigned different weights according to the deviation degrees between this feature of all objects and that of a certain cluster prototype. A series of experiments on a synthetic dataset and several real-world datasets are implemented to verify the effectiveness and efficiency of the proposal.

scholarly journals Adaptively Multi-Objective Adversarial Training for Dialogue Generation

2020 ◽  
Author(s):  
Xuemiao Zhang ◽  
Zhouxing Tan ◽  
Xiaoning Zhang ◽  
Yang Cao ◽  
Rui Yan
Keyword(s):  
Real World ◽  
Optimization Problem ◽  
Experimental Results ◽  
Sampling Distribution ◽  
Multi Objective Optimization ◽  
Generation Task ◽  
Multi Objective ◽  
Adversarial Models ◽  
Adversarial Training ◽  
Real World Datasets

Naive neural dialogue generation models tend to produce repetitive and dull utterances. The promising adversarial models train the generator against a well-designed discriminator to push it to improve towards the expected direction. However, assessing dialogues requires consideration of many aspects of linguistics, which are difficult to be fully covered by a single discriminator. To address it, we reframe the dialogue generation task as a multi-objective optimization problem and propose a novel adversarial dialogue generation framework with multiple discriminators that excel in different objectives for multiple linguistic aspects, called AMPGAN, whose feasibility is proved by theoretical derivations. Moreover, we design an adaptively adjusted sampling distribution to balance the discriminators and promote the overall improvement of the generator by continuing to focus on these objectives that the generator is not performing well relatively. Experimental results on two real-world datasets show a significant improvement over the baselines.

Sign in / Sign up

Export Citation Format

Share Document