Sufficient Dimension Reduction via Direct Estimation of the Gradients of Logarithmic Conditional Densities

2018 ◽  
Vol 30 (2) ◽  
pp. 477-504 ◽  
Author(s):  
Hiroaki Sasaki ◽  
Voot Tangkaratt ◽  
Gang Niu ◽  
Masashi Sugiyama
Keyword(s):  
Convergence Rate ◽  
Dimension Reduction ◽  
Computational Efficiency ◽  
Low Rank ◽  
Conditional Density ◽  
Data Sets ◽  
Sufficient Dimension Reduction ◽  
Output Data ◽  
Outer Product ◽  
Benchmark Data

Sufficient dimension reduction (SDR) is aimed at obtaining the low-rank projection matrix in the input space such that information about output data is maximally preserved. Among various approaches to SDR, a promising method is based on the eigendecomposition of the outer product of the gradient of the conditional density of output given input. In this letter, we propose a novel estimator of the gradient of the logarithmic conditional density that directly fits a linear-in-parameter model to the true gradient under the squared loss. Thanks to this simple least-squares formulation, its solution can be computed efficiently in a closed form. Then we develop a new SDR method based on the proposed gradient estimator. We theoretically prove that the proposed gradient estimator, as well as the SDR solution obtained from it, achieves the optimal parametric convergence rate. Finally, we experimentally demonstrate that our SDR method compares favorably with existing approaches in both accuracy and computational efficiency on a variety of artificial and benchmark data sets.

scholarly journals Sufficient Dimension Reduction via Squared-Loss Mutual Information Estimation

2013 ◽  
Vol 25 (3) ◽  
pp. 725-758 ◽  
Author(s):  
Taiji Suzuki ◽  
Masashi Sugiyama
Keyword(s):  
Mutual Information ◽  
Convergence Rate ◽  
Dimension Reduction ◽  
Density Ratio ◽  
Dimensional Subspace ◽  
Parametric Models ◽  
Gradient Algorithm ◽  
Ratio Estimator ◽  
Sufficient Dimension Reduction ◽  
Nonparametric Models

The goal of sufficient dimension reduction in supervised learning is to find the low-dimensional subspace of input features that contains all of the information about the output values that the input features possess. In this letter, we propose a novel sufficient dimension-reduction method using a squared-loss variant of mutual information as a dependency measure. We apply a density-ratio estimator for approximating squared-loss mutual information that is formulated as a minimum contrast estimator on parametric or nonparametric models. Since cross-validation is available for choosing an appropriate model, our method does not require any prespecified structure on the underlying distributions. We elucidate the asymptotic bias of our estimator on parametric models and the asymptotic convergence rate on nonparametric models. The convergence analysis utilizes the uniform tail-bound of a U-process, and the convergence rate is characterized by the bracketing entropy of the model. We then develop a natural gradient algorithm on the Grassmann manifold for sufficient subspace search. The analytic formula of our estimator allows us to compute the gradient efficiently. Numerical experiments show that the proposed method compares favorably with existing dimension-reduction approaches on artificial and benchmark data sets.

Non-Covalent Interactions Atlas Benchmark Data Sets 3: Repulsive Contacts

2021 ◽  
Vol 17 (3) ◽  
pp. 1548-1561
Author(s):  
Kristian Kříž ◽  
Martin Nováček ◽  
Jan Řezáč
Keyword(s):  
Data Sets ◽  
Benchmark Data ◽  
Non Covalent Interactions ◽  
Covalent Interactions

scholarly journals Cumulative Median Estimation for Sufficient Dimension Reduction

Stats ◽  
2021 ◽  
Vol 4 (1) ◽  
pp. 138-145
Author(s):  
Stephen Babos ◽  
Andreas Artemiou
Keyword(s):  
Dimension Reduction ◽  
Real Data ◽  
Sufficient Dimension Reduction

In this paper, we present the Cumulative Median Estimation (CUMed) algorithm for robust sufficient dimension reduction. Compared with non-robust competitors, this algorithm performs better when there are outliers present in the data and comparably when outliers are not present. This is demonstrated in simulated and real data experiments.

scholarly journals Clipped Matrix Completion: A Remedy for Ceiling Effects

2019 ◽  
Vol 33 ◽  
pp. 5151-5158
Author(s):  
Takeshi Teshima ◽  
Miao Xu ◽  
Issei Sato ◽  
Masashi Sugiyama
Keyword(s):  
Matrix Completion ◽  
Low Rank ◽  
Trace Norm ◽  
Minimization Algorithm ◽  
Benchmark Data ◽  
Hinge Loss ◽  
Norm Minimization ◽  
Rank Matrix ◽  
Recovery Error ◽  
Low Rank Matrix

We consider the problem of recovering a low-rank matrix from its clipped observations. Clipping is conceivable in many scientific areas that obstructs statistical analyses. On the other hand, matrix completion (MC) methods can recover a low-rank matrix from various information deficits by using the principle of low-rank completion. However, the current theoretical guarantees for low-rank MC do not apply to clipped matrices, as the deficit depends on the underlying values. Therefore, the feasibility of clipped matrix completion (CMC) is not trivial. In this paper, we first provide a theoretical guarantee for the exact recovery of CMC by using a trace-norm minimization algorithm. Furthermore, we propose practical CMC algorithms by extending ordinary MC methods. Our extension is to use the squared hinge loss in place of the squared loss for reducing the penalty of overestimation on clipped entries. We also propose a novel regularization term tailored for CMC. It is a combination of two trace-norm terms, and we theoretically bound the recovery error under the regularization. We demonstrate the effectiveness of the proposed methods through experiments using both synthetic and benchmark data for recommendation systems.

Bayesian Trigonometric Support Vector Classifier

2003 ◽  
Vol 15 (9) ◽  
pp. 2227-2254 ◽  
Author(s):  
Wei Chu ◽  
S. Sathiya Keerthi ◽  
Chong Jin Ong
Keyword(s):  
Loss Function ◽  
Gaussian Processes ◽  
Likelihood Function ◽  
Support Vector ◽  
Data Sets ◽  
Model Adaptation ◽  
Bayesian Techniques ◽  
Benchmark Data ◽  
Support Vector Classifier ◽  
Set Up

This letter describes Bayesian techniques for support vector classification. In particular, we propose a novel differentiable loss function, called the trigonometric loss function, which has the desirable characteristic of natural normalization in the likelihood function, and then follow standard gaussian processes techniques to set up a Bayesian framework. In this framework, Bayesian inference is used to implement model adaptation, while keeping the merits of support vector classifier, such as sparseness and convex programming. This differs from standard gaussian processes for classification. Moreover, we put forward class probability in making predictions. Experimental results on benchmark data sets indicate the usefulness of this approach.

scholarly journals AA-HMM: An Anti-Adversarial Hidden Markov Model for Network-Based Intrusion Detection

2018 ◽  
Vol 8 (12) ◽  
pp. 2421 ◽  
Author(s):  
Chongya Song ◽  
Alexander Pons ◽  
Kang Yen
Keyword(s):  
Markov Model ◽  
Hidden Markov Model ◽  
Performance Metrics ◽  
Hidden Markov ◽  
Data Sets ◽  
Learning Abilities ◽  
Benchmark Data ◽  
Malicious Behavior ◽  
Network Intrusion ◽  
The Common

In the field of network intrusion, malware usually evades anomaly detection by disguising malicious behavior as legitimate access. Therefore, detecting these attacks from network traffic has become a challenge in this an adversarial setting. In this paper, an enhanced Hidden Markov Model, called the Anti-Adversarial Hidden Markov Model (AA-HMM), is proposed to effectively detect evasion pattern, using the Dynamic Window and Threshold techniques to achieve adaptive, anti-adversarial, and online-learning abilities. In addition, a concept called Pattern Entropy is defined and acts as the foundation of AA-HMM. We evaluate the effectiveness of our approach employing two well-known benchmark data sets, NSL-KDD and CTU-13, in terms of the common performance metrics and the algorithm’s adaptation and anti-adversary abilities.

Sign in / Sign up

Export Citation Format

Share Document