One-Shot Neural Architecture Search by Dynamically Pruning Supernet in Hierarchical Order

2021 ◽  
pp. 2150029
Author(s):  
Jianwei Zhang ◽  
Dong Li ◽  
Lituan Wang ◽  
Lei Zhang
Keyword(s):  
State Of The Art ◽  
Search Space ◽  
Experimental Results ◽  
Research Interest ◽  
Search Process ◽  
Search Efficiency ◽  
Neural Architecture ◽  
Speed Up ◽  
The One

Neural Architecture Search (NAS), which aims at automatically designing neural architectures, recently draw a growing research interest. Different from conventional NAS methods, in which a large number of neural architectures need to be trained for evaluation, the one-shot NAS methods only have to train one supernet which synthesizes all the possible candidate architectures. As a result, the search efficiency could be significantly improved by sharing the supernet’s weights during the candidate architectures’ evaluation. This strategy could greatly speed up the search process but suffer a challenge that the evaluation based on sharing weights is not predictive enough. Recently, pruning the supernet during the search has been proven to be an efficient way to alleviate this problem. However, the pruning direction in complex-structured search space remains unexplored. In this paper, we revisited the role of path dropout strategy, which drops the neural operations instead of the neurons, in supernet training, and several interesting characters of the supernet trained with dropout are found. Based on the observations, a Hierarchically-Ordered Pruning Neural Architecture Search (HOPNAS) algorithm is proposed by dynamically pruning the supernet with a proper pruning direction. Experimental results indicate that our method is competitive with state-of-the-art approaches on CIFAR10 and ImageNet.

Evaluating Top-k Skyline Queries Efficiently

2011 ◽  
pp. 102-117
Author(s):  
Marlene Goncalves ◽  
María Esther Vidal
Keyword(s):  
State Of The Art ◽  
Search Space ◽  
Large Datasets ◽  
The State ◽  
Experimental Results ◽  
High Dimensional ◽  
Skyline Queries ◽  
Speed Up

Criteria that induce a Skyline naturally represent user’s preference conditions useful to discard irrelevant data in large datasets. However, in the presence of high-dimensional Skyline spaces, the size of the Skyline can still be very large. To identify the best k points among the Skyline, the Top-k Skyline approach has been proposed. This chapter describes existing solutions and proposes to use the TKSI algorithm for the Top-k Skyline problem. TKSI reduces the search space by computing only a subset of the Skyline that is required to produce the top-k objects. In addition, the Skyline Frequency Metric is implemented to discriminate among the Skyline objects those that best meet the multidimensional criteria. This chapter’s authors have empirically studied the quality of TKSI, and their experimental results show the TKSI may be able to speed up the computation of the Top-k Skyline in at least 50% percent with regard to the state-of-the-art solutions.

scholarly journals Neural Architecture Search for a Highly Efficient Network with Random Skip Connections

2020 ◽  
Vol 10 (11) ◽  
pp. 3712
Author(s):  
Dongjing Shan ◽  
Xiongwei Zhang ◽  
Wenhua Shi ◽  
Li Li
Keyword(s):  
State Of The Art ◽  
Cell Structure ◽  
Frequency Discrimination ◽  
Search Space ◽  
Short Term ◽  
Cell Parameters ◽  
Neural Architecture ◽  
Proposed Model ◽  
Initialization Scheme

Regarding the sequence learning of neural networks, there exists a problem of how to capture long-term dependencies and alleviate the gradient vanishing phenomenon. To manage this problem, we proposed a neural network with random connections via a scheme of a neural architecture search. First, a dense network was designed and trained to construct a search space, and then another network was generated by random sampling in the space, whose skip connections could transmit information directly over multiple periods and capture long-term dependencies more efficiently. Moreover, we devised a novel cell structure that required less memory and computational power than the structures of long short-term memories (LSTMs), and finally, we performed a special initialization scheme on the cell parameters, which could permit unhindered gradient propagation on the time axis at the beginning of training. In the experiments, we evaluated four sequential tasks: adding, copying, frequency discrimination, and image classification; we also adopted several state-of-the-art methods for comparison. The experimental results demonstrated that our proposed model achieved the best performance.

scholarly journals Burnout: de stand van zaken

2013 ◽  
Vol 29 (3) ◽  
Author(s):  
Toon Taris ◽  
Irene Houtman ◽  
Wilmar Schaufeli
Keyword(s):  
Work Performance ◽  
Subjective Experience ◽  
State Of The Art ◽  
The State ◽  
Related Phenomenon ◽  
Related Factors ◽  
Work Related ◽  
Theoretical Perspectives ◽  
The One

Burnout: the state of the art Burnout: the state of the art This manuscript presents an overview of the state-of-the-art in burnout research. Burnout is a work-related syndrome of extreme fatigue (exhaustion), distancing from work (cynicism), and low levels of professional efficacy, that is rooted in either the characteristics of one’s job or on individual’s way of coping with these characteristics. Being a work-related phenomenon, burnout can be distinguished from other, more general mental illness such as depression. Further, burnout and engagement can be distinguished as well.Burnout can be assessed by asking workers about their personal and subjective experience of their health. Objective ways of measuring burnout (e.g., using psychophysiological measures) have as yet not been useful. The emphasis on burnout as a work-related phenomenon is also evident from current theoretical perspectives. On the one hand, these perspectives highlight the role of work-related factors, whereas on the other hand these perspectives focus on the role of personality characteristics such as neuroticism and (over-)commitment. Burnout affects the organization as well as individual workers; it is related to elevated levels of sickness absence, a higher risk of work disability, and a lower level of work performance. Both person-directed and organization-directed interventions may affect burnout positively.

scholarly journals Efficient exploratory clustering analyses in large-scale exploration processes

2021 ◽  
Author(s):  
Manuel Fritz ◽  
Michael Behringer ◽  
Dennis Tschechlov ◽  
Holger Schwarz
Keyword(s):  
Large Scale ◽  
Clustering Algorithm ◽  
Comprehensive Evaluation ◽  
State Of The Art ◽  
Clustering Algorithms ◽  
Search Space ◽  
Large Datasets ◽  
Search Spaces ◽  
Multiple Challenges ◽  
The One

AbstractClustering is a fundamental primitive in manifold applications. In order to achieve valuable results in exploratory clustering analyses, parameters of the clustering algorithm have to be set appropriately, which is a tremendous pitfall. We observe multiple challenges for large-scale exploration processes. On the one hand, they require specific methods to efficiently explore large parameter search spaces. On the other hand, they often exhibit large runtimes, in particular when large datasets are analyzed using clustering algorithms with super-polynomial runtimes, which repeatedly need to be executed within exploratory clustering analyses. We address these challenges as follows: First, we present LOG-Means and show that it provides estimates for the number of clusters in sublinear time regarding the defined search space, i.e., provably requiring less executions of a clustering algorithm than existing methods. Second, we demonstrate how to exploit fundamental characteristics of exploratory clustering analyses in order to significantly accelerate the (repetitive) execution of clustering algorithms on large datasets. Third, we show how these challenges can be tackled at the same time. To the best of our knowledge, this is the first work which simultaneously addresses the above-mentioned challenges. In our comprehensive evaluation, we unveil that our proposed methods significantly outperform state-of-the-art methods, thus especially supporting novice analysts for exploratory clustering analyses in large-scale exploration processes.

scholarly journals Loss-based active learning via double-branch deep network

2021 ◽  
Vol 18 (5) ◽  
pp. 172988142110449
Author(s):  
Qiang Fang ◽  
Xin Xu ◽  
Dengqing Tang
Keyword(s):  
Active Learning ◽  
Visual Recognition ◽  
State Of The Art ◽  
Experimental Results ◽  
Research Interest ◽  
Selection Strategy ◽  
Network Architectures ◽  
Data Annotation ◽  
Deep Network ◽  
Effectiveness And Efficiency

Due to the limitation of data annotation and the ability of dealing with label-efficient problems, active learning has received lots of research interest in recent years. Most of the existing approaches focus on designing a different selection strategy to achieve better performance for special tasks; however, the performance of the strategy still needs to be improved. In this work, we focus on improving the performance of active learning and propose a loss-based strategy that learns to predict target losses of unlabeled inputs to select the most uncertain samples, which is designed to learn a better selection strategy based on a double-branch deep network. Experimental results on two visual recognition tasks show that our approach achieves the state-of-the-art performance compared with previous methods. Moreover, our approach is also robust to different network architectures, biased initial labels, noisy oracles, or sampling budget sizes, and the complexity is also competitive, which demonstrates the effectiveness and efficiency of our proposed approach.

scholarly journals TextNAS: A Neural Architecture Search Space Tailored for Text Representation

2020 ◽  
Vol 34 (05) ◽  
pp. 9242-9249
Author(s):  
Yujing Wang ◽  
Yaming Yang ◽  
Yiren Chen ◽  
Jing Bai ◽  
Ce Zhang ◽  
...  
Keyword(s):  
Text Classification ◽  
Network Architecture ◽  
State Of The Art ◽  
Search Space ◽  
Search Algorithms ◽  
Text Representation ◽  
Neural Architecture ◽  
Good Potential ◽  
Automatic Search ◽  
Public Datasets

Learning text representation is crucial for text classification and other language related tasks. There are a diverse set of text representation networks in the literature, and how to find the optimal one is a non-trivial problem. Recently, the emerging Neural Architecture Search (NAS) techniques have demonstrated good potential to solve the problem. Nevertheless, most of the existing works of NAS focus on the search algorithms and pay little attention to the search space. In this paper, we argue that the search space is also an important human prior to the success of NAS in different applications. Thus, we propose a novel search space tailored for text representation. Through automatic search, the discovered network architecture outperforms state-of-the-art models on various public datasets on text classification and natural language inference tasks. Furthermore, some of the design principles found in the automatic network agree well with human intuition.

Theoretical Convergence Guarantees for Cooperative Coevolutionary Algorithms

2010 ◽  
Vol 18 (4) ◽  
pp. 581-615 ◽  
Author(s):  
Liviu Panait
Keyword(s):  
Theoretical Foundation ◽  
Formal Model ◽  
Optimal Solution ◽  
Theoretical Work ◽  
Search Space ◽  
Simple Problem ◽  
Search Process ◽  
Coevolutionary Algorithms ◽  
Suboptimal Solutions ◽  
Speed Up

Cooperative coevolutionary algorithms have the potential to significantly speed up the search process by dividing the space into parts that can each be conquered separately. However, recent research presented theoretical and empirical arguments that these algorithms tend to converge to suboptimal solutions in the search space, and are thus not fit for optimization tasks. This paper details an extended formal model for cooperative coevolutionary algorithms, and uses it to explore possible reasons these algorithms converge to optimal or suboptimal solutions. We demonstrate that, under specific conditions, this theoretical model will converge to the globally optimal solution. The proofs provide the underlying theoretical foundation for a better application of cooperative coevolutionary algorithms. We demonstrate the practical advantages of applying ideas from this theoretical work to a simple problem domain.

scholarly journals A Method for Gradient Differentiable Network Architecture Search by Selecting and Clustering Candidate Operations

2021 ◽  
Vol 11 (23) ◽  
pp. 11436
Author(s):  
Ha Yoon Song
Keyword(s):  
Deep Learning ◽  
Network Architecture ◽  
Gradient Descent ◽  
Selection Process ◽  
Search Space ◽  
Search Cost ◽  
Second Stage ◽  
Neural Architecture ◽  
Speed Up ◽  
Mixed Operation

The current evolution of deep learning requires further optimization in terms of accuracy and time. From the perspective of new requirements, AutoML is an area that could provide possible solutions. AutoML has a neural architecture search (NAS) field. DARTS is a widely used approach in NAS and is based on gradient descent; however, it has some drawbacks. In this study, we attempted to overcome some of the drawbacks of DARTS by improving the accuracy and decreasing the search cost. The DARTS algorithm uses a mixed operation that combines all operations in the search space. The architecture parameter of each operation comprising a mixed operation is trained using gradient descent, and the operation with the largest architecture parameter is selected. The use of a mixed operation causes a problem called vote dispersion: similar operations share architecture parameters during gradient descent; thus, there are cases where the most important operation is disregarded. In this selection process, vote dispersion causes DARTS performance to degrade. To cope with this problem, we propose a new algorithm based on DARTS called DG-DARTS. Two search stages are introduced, and the clustering of operations is applied in DG-DARTS. In summary, DG-DARTS achieves an error rate of 2.51% on the CIFAR10 dataset, and its search cost is 0.2 GPU days because the search space of the second stage is reduced by half. The speed-up factor of DG-DARTS to DARTS is 6.82, which indicates that the search cost of DG-DARTS is only 13% that of DARTS.

DISCRIMINATIVELY WEIGHTED NAIVE BAYES AND ITS APPLICATION IN TEXT CLASSIFICATION

2012 ◽  
Vol 21 (01) ◽  
pp. 1250007 ◽  
Author(s):  
LIANGXIAO JIANG ◽  
DIANHONG WANG ◽  
ZHIHUA CAI
Keyword(s):  
Naive Bayes ◽  
State Of The Art ◽  
Naïve Bayes ◽  
Experimental Results ◽  
Data Sets ◽  
Conditional Independence Assumption ◽  
Instance Weighting ◽  
Text Classifiers ◽  
Bayes Algorithm ◽  
The One

Many approaches are proposed to improve naive Bayes by weakening its conditional independence assumption. In this paper, we work on the approach of instance weighting and propose an improved naive Bayes algorithm by discriminative instance weighting. We called it Discriminatively Weighted Naive Bayes. In each iteration of it, different training instances are discriminatively assigned different weights according to the estimated conditional probability loss. The experimental results based on a large number of UCI data sets validate its effectiveness in terms of the classification accuracy and AUC. Besides, the experimental results on the running time show that our Discriminatively Weighted Naive Bayes performs almost as efficiently as the state-of-the-art Discriminative Frequency Estimate learning method, and significantly more efficient than Boosted Naive Bayes. At last, we apply the idea of discriminatively weighted learning in our algorithm to some state-of-the-art naive Bayes text classifiers, such as multinomial naive Bayes, complement naive Bayes and the one-versus-all-but-one model, and have achieved remarkable improvements.

scholarly journals Learning the Structure of Bayesian Networks: A Quantitative Assessment of the Effect of Different Algorithmic Schemes

Complexity ◽  
2018 ◽  
Vol 2018 ◽  
pp. 1-12 ◽  
Author(s):  
Stefano Beretta ◽  
Mauro Castelli ◽  
Ivo Gonçalves ◽  
Roberto Henriques ◽  
Daniele Ramazzotti
Keyword(s):  
Bayesian Networks ◽  
Quantitative Assessment ◽  
State Of The Art ◽  
Simulated Data ◽  
Detailed Comparison ◽  
Search Space ◽  
Continuous Variables ◽  
Challenging Tasks ◽  
The One ◽  
Algorithmic Approaches

One of the most challenging tasks when adopting Bayesian networks (BNs) is the one of learning their structure from data. This task is complicated by the huge search space of possible solutions and by the fact that the problem is NP-hard. Hence, a full enumeration of all the possible solutions is not always feasible and approximations are often required. However, to the best of our knowledge, a quantitative analysis of the performance and characteristics of the different heuristics to solve this problem has never been done before. For this reason, in this work, we provide a detailed comparison of many different state-of-the-art methods for structural learning on simulated data considering both BNs with discrete and continuous variables and with different rates of noise in the data. In particular, we investigate the performance of different widespread scores and algorithmic approaches proposed for the inference and the statistical pitfalls within them.

Sign in / Sign up

Export Citation Format

Share Document