scholarly journals An Empirical Evaluation of Ranking Measures With Respect to Robustness to Noise

2014 ◽  
Vol 49 ◽  
pp. 241-267 ◽  
Author(s):  
D. Berrar
Keyword(s):  
Empirical Evaluation ◽  
Data Sets ◽  
Real World Data ◽  
Roc Convex Hull ◽  
Different Types ◽  
Kolmogorov Smirnov ◽  
Robustness To Noise ◽  
Ranking Measures ◽  
Robust To Noise ◽  
Smirnov Statistic

Ranking measures play an important role in model evaluation and selection. Using both synthetic and real-world data sets, we investigate how different types and levels of noise affect the area under the ROC curve (AUC), the area under the ROC convex hull, the scored AUC, the Kolmogorov-Smirnov statistic, and the H-measure. In our experiments, the AUC was, overall, the most robust among these measures, thereby reinvigorating it as a reliable metric despite its well-known deficiencies. This paper also introduces a novel ranking measure, which is remarkably robust to noise yet conceptually simple.

Direction-dependence analysis

2015 ◽  
Vol 39 (6) ◽  
pp. 570-580 ◽  
Author(s):  
Wolfgang Wiedermann ◽  
Alexander von Eye
Keyword(s):  
Real World ◽  
Empirical Evaluation ◽  
Data Sets ◽  
Dependence Analysis ◽  
Linear Regression Models ◽  
Real World Data ◽  
Variable Pair ◽  
Causal Search ◽  
Direction Dependence ◽  
New Guidelines

The concept of direction dependence has attracted growing attention due to its potential to help decide which of two competing linear regression models ( X → Y or Y → X) is more likely to reflect the correct causal flow. Several tests have been proposed to evaluate hypotheses compatible with direction dependence. In this issue, Thoemmes (2015) reports results of an empirical evaluation of direction-dependence tests using real-world data sets with known causal ordering and concludes that the tests (known to perform excellent in simulation studies) perform poorly in the real-world setting. The present article aims at answering the question how this is possible. First, we review potential conceptual issues associated with Thoemmes’ (2015) approach. We argue that direction dependence is best conceptualized as a confirmatory approach to test focused directional theories. Thoemmes’ (2015) evaluation is based on an exploratory use of direction dependence. It implicitly follows the tradition of causal search algorithms. Second, we discuss potential statistical issues associated with Thoemmes’ (2015) selection schemes used to decide whether a variable pair is suitable for direction-dependence analysis. Based on these issues, new tests of direction dependence as well as new guidelines for confirmatory direction-dependence analysis are proposed. An empirical example is given to illustrate the application of these guidelines.

scholarly journals Analyzing the impact of missing values and selection bias on fairness

2021 ◽  
Author(s):  
Yanchen Wang ◽  
Lisa Singh
Keyword(s):  
Selection Bias ◽  
Real World ◽  
Missing Values ◽  
Empirical Evaluation ◽  
Data Sets ◽  
Generation Process ◽  
Real World Data ◽  
World Data ◽  
Impact Prediction ◽  
The Impact

AbstractAlgorithmic decision making is becoming more prevalent, increasingly impacting people’s daily lives. Recently, discussions have been emerging about the fairness of decisions made by machines. Researchers have proposed different approaches for improving the fairness of these algorithms. While these approaches can help machines make fairer decisions, they have been developed and validated on fairly clean data sets. Unfortunately, most real-world data have complexities that make them more dirty. This work considers two of these complexities by analyzing the impact of two real-world data issues on fairness—missing values and selection bias—for categorical data. After formulating this problem and showing its existence, we propose fixing algorithms for data sets containing missing values and/or selection bias that use different forms of reweighting and resampling based upon the missing value generation process. We conduct an extensive empirical evaluation on both real-world and synthetic data using various fairness metrics, and demonstrate how different missing values generated from different mechanisms and selection bias impact prediction fairness, even when prediction accuracy remains fairly constant.

scholarly journals Multi-Label Causal Feature Selection

2020 ◽  
Vol 34 (04) ◽  
pp. 6430-6437 ◽  
Author(s):  
Xingyu Wu ◽  
Bingbing Jiang ◽  
Kui Yu ◽  
Huanhuan Chen ◽  
Chunyan Miao
Keyword(s):  
Feature Selection ◽  
State Of The Art ◽  
Causal Mechanism ◽  
Data Sets ◽  
Causal Relationships ◽  
Real World Data ◽  
Data Set ◽  
Complete Representation ◽  
The Past ◽  
Different Types

Multi-label feature selection has received considerable attentions during the past decade. However, existing algorithms do not attempt to uncover the underlying causal mechanism, and individually solve different types of variable relationships, ignoring the mutual effects between them. Furthermore, these algorithms lack of interpretability, which can only select features for all labels, but cannot explain the correlation between a selected feature and a certain label. To address these problems, in this paper, we theoretically study the causal relationships in multi-label data, and propose a novel Markov blanket based multi-label causal feature selection (MB-MCF) algorithm. MB-MCF mines the causal mechanism of labels and features first, to obtain a complete representation of information about labels. Based on the causal relationships, MB-MCF then selects predictive features and simultaneously distinguishes common features shared by multiple labels and label-specific features owned by single labels. Experiments on real-world data sets validate that MB-MCF could automatically determine the number of selected features and simultaneously achieve the best performance compared with state-of-the-art methods. An experiment in Emotions data set further demonstrates the interpretability of MB-MCF.

TrustSVD: A Novel Trust-Based Matrix Factorization Model with User Trust and Item Ratings

2017 ◽  
Vol 7 (11) ◽  
pp. 7 ◽  
Author(s):  
K Sobha Rani
Keyword(s):  
Matrix Factorization ◽  
Social Trust ◽  
State Of The Art ◽  
Data Sets ◽  
Real World Data ◽  
Recommendation Algorithm ◽  
Active User ◽  
Factorization Model ◽  
The Social ◽  
Matrix Factorization Technique

Collaborative filtering suffers from the problems of data sparsity and cold start, which dramatically degrade recommendation performance. To help resolve these issues, we propose TrustSVD, a trust-based matrix factorization technique. By analyzing the social trust data from four real-world data sets, we conclude that not only the explicit but also the implicit influence of both ratings and trust should be taken into consideration in a recommendation model. Hence, we build on top of a state-of-the-art recommendation algorithm SVD++ which inherently involves the explicit and implicit influence of rated items, by further incorporating both the explicit and implicit influence of trusted users on the prediction of items for an active user. To our knowledge, the work reported is the first to extend SVD++ with social trust information. Experimental results on the four data sets demonstrate that our approach TrustSVD achieves better accuracy than other ten counterparts, and can better handle the concerned issues.

scholarly journals Hfinger: Malware HTTP Request Fingerprinting

Entropy ◽  
2021 ◽  
Vol 23 (5) ◽  
pp. 507
Author(s):  
Piotr Białczak ◽  
Wojciech Mazurczyk
Keyword(s):  
Real World ◽  
Network Traffic ◽  
Experimental Evaluation ◽  
Data Sets ◽  
Real World Data ◽  
Malicious Software ◽  
Default Mode ◽  
World Data ◽  
Effectiveness Analysis ◽  
Http Protocol

Malicious software utilizes HTTP protocol for communication purposes, creating network traffic that is hard to identify as it blends into the traffic generated by benign applications. To this aim, fingerprinting tools have been developed to help track and identify such traffic by providing a short representation of malicious HTTP requests. However, currently existing tools do not analyze all information included in the HTTP message or analyze it insufficiently. To address these issues, we propose Hfinger, a novel malware HTTP request fingerprinting tool. It extracts information from the parts of the request such as URI, protocol information, headers, and payload, providing a concise request representation that preserves the extracted information in a form interpretable by a human analyst. For the developed solution, we have performed an extensive experimental evaluation using real-world data sets and we also compared Hfinger with the most related and popular existing tools such as FATT, Mercury, and p0f. The conducted effectiveness analysis reveals that on average only 1.85% of requests fingerprinted by Hfinger collide between malware families, what is 8–34 times lower than existing tools. Moreover, unlike these tools, in default mode, Hfinger does not introduce collisions between malware and benign applications and achieves it by increasing the number of fingerprints by at most 3 times. As a result, Hfinger can effectively track and hunt malware by providing more unique fingerprints than other standard tools.

Learning emotional word embeddings for sentiment analysis

2021 ◽  
pp. 1-13
Author(s):  
Qingtian Zeng ◽  
Xishi Zhao ◽  
Xiaohui Hu ◽  
Hua Duan ◽  
Zhongying Zhao ◽  
...  
Keyword(s):  
Sentiment Analysis ◽  
Language Processing ◽  
State Of The Art ◽  
Research Problem ◽  
Emotional Word ◽  
Classification Model ◽  
Data Sets ◽  
Word Embeddings ◽  
Real World Data ◽  
Text Documents

Word embeddings have been successfully applied in many natural language processing tasks due to its their effectiveness. However, the state-of-the-art algorithms for learning word representations from large amounts of text documents ignore emotional information, which is a significant research problem that must be addressed. To solve the above problem, we propose an emotional word embedding (EWE) model for sentiment analysis in this paper. This method first applies pre-trained word vectors to represent document features using two different linear weighting methods. Then, the resulting document vectors are input to a classification model and used to train a text sentiment classifier, which is based on a neural network. In this way, the emotional polarity of the text is propagated into the word vectors. The experimental results on three kinds of real-world data sets demonstrate that the proposed EWE model achieves superior performances on text sentiment prediction, text similarity calculation, and word emotional expression tasks compared to other state-of-the-art models.

scholarly journals Different algorithms, different models

2021 ◽  
Author(s):  
Martyna Daria Swiatczak
Keyword(s):  
Comparative Analysis ◽  
Real World ◽  
Qualitative Comparative Analysis ◽  
Comparative Methods ◽  
Data Sets ◽  
Simulation Studies ◽  
Threshold Values ◽  
Real World Data ◽  
Software Packages ◽  
Methodological Approaches

AbstractThis study assesses the extent to which the two main Configurational Comparative Methods (CCMs), i.e. Qualitative Comparative Analysis (QCA) and Coincidence Analysis (CNA), produce different models. It further explains how this non-identity is due to the different algorithms upon which both methods are based, namely QCA’s Quine–McCluskey algorithm and the CNA algorithm. I offer an overview of the fundamental differences between QCA and CNA and demonstrate both underlying algorithms on three data sets of ascending proximity to real-world data. Subsequent simulation studies in scenarios of varying sample sizes and degrees of noise in the data show high overall ratios of non-identity between the QCA parsimonious solution and the CNA atomic solution for varying analytical choices, i.e. different consistency and coverage threshold values and ways to derive QCA’s parsimonious solution. Clarity on the contrasts between the two methods is supposed to enable scholars to make more informed decisions on their methodological approaches, enhance their understanding of what is happening behind the results generated by the software packages, and better navigate the interpretation of results. Clarity on the non-identity between the underlying algorithms and their consequences for the results is supposed to provide a basis for a methodological discussion about which method and which variants thereof are more successful in deriving which search target.

scholarly journals A Hypothesis Test for the Goodness-of-Fit of the Marginal Distribution of a Time Series with Application to Stablecoin Data

2021 ◽  
Vol 5 (1) ◽  
pp. 10
Author(s):  
Mark Levene
Keyword(s):  
Time Series ◽  
Goodness Of Fit ◽  
Marginal Distribution ◽  
Hypothesis Test ◽  
Data Sets ◽  
Test Statistic ◽  
Sample Test ◽  
Kolmogorov Smirnov ◽  
Heavy Tailed ◽  
Jensen Shannon Divergence

A bootstrap-based hypothesis test of the goodness-of-fit for the marginal distribution of a time series is presented. Two metrics, the empirical survival Jensen–Shannon divergence (ESJS) and the Kolmogorov–Smirnov two-sample test statistic (KS2), are compared on four data sets—three stablecoin time series and a Bitcoin time series. We demonstrate that, after applying first-order differencing, all the data sets fit heavy-tailed α-stable distributions with 1<α<2 at the 95% confidence level. Moreover, ESJS is more powerful than KS2 on these data sets, since the widths of the derived confidence intervals for KS2 are, proportionately, much larger than those of ESJS.

Auto-sharing parameters for transfer learning based on multi-objective optimization

2021 ◽  
pp. 1-13
Author(s):  
Hailin Liu ◽  
Fangqing Gu ◽  
Zixian Lin
Keyword(s):  
Transfer Learning ◽  
Optimization Problem ◽  
Data Sets ◽  
Multi Objective Optimization ◽  
Particle Swarm Optimizer ◽  
Real World Data ◽  
Data Set ◽  
Target Task ◽  
Main Research ◽  
Multi Objective

Transfer learning methods exploit similarities between different datasets to improve the performance of the target task by transferring knowledge from source tasks to the target task. “What to transfer” is a main research issue in transfer learning. The existing transfer learning method generally needs to acquire the shared parameters by integrating human knowledge. However, in many real applications, an understanding of which parameters can be shared is unknown beforehand. Transfer learning model is essentially a special multi-objective optimization problem. Consequently, this paper proposes a novel auto-sharing parameter technique for transfer learning based on multi-objective optimization and solves the optimization problem by using a multi-swarm particle swarm optimizer. Each task objective is simultaneously optimized by a sub-swarm. The current best particle from the sub-swarm of the target task is used to guide the search of particles of the source tasks and vice versa. The target task and source task are jointly solved by sharing the information of the best particle, which works as an inductive bias. Experiments are carried out to evaluate the proposed algorithm on several synthetic data sets and two real-world data sets of a school data set and a landmine data set, which show that the proposed algorithm is effective.

Sign in / Sign up

Export Citation Format

Share Document