scholarly journals Progressive Automatic Design of Search Space for One-Shot Neural Architecture Search

2022 ◽  
pp. 91-110
Author(s):  
Xin Xia ◽  
Xuefeng Xiao ◽  
Xing Wang
Keyword(s):  
Search Space ◽  
Automatic Design ◽  
Neural Architecture

A Comprehensive Survey of Neural Architecture Search

2021 ◽  
Vol 54 (4) ◽  
pp. 1-34
Author(s):  
Pengzhen Ren ◽  
Yun Xiao ◽  
Xiaojun Chang ◽  
Po-yao Huang ◽  
Zhihui Li ◽  
...  
Keyword(s):  
Research Work ◽  
Search Space ◽  
Feature Representation ◽  
Future Research ◽  
Related Research ◽  
Neural Architecture ◽  
Comprehensive Survey ◽  
Future Research Directions ◽  
Intuitive Idea ◽  
New Perspective

Deep learning has made substantial breakthroughs in many fields due to its powerful automatic representation capabilities. It has been proven that neural architecture design is crucial to the feature representation of data and the final performance. However, the design of the neural architecture heavily relies on the researchers’ prior knowledge and experience. And due to the limitations of humans’ inherent knowledge, it is difficult for people to jump out of their original thinking paradigm and design an optimal model. Therefore, an intuitive idea would be to reduce human intervention as much as possible and let the algorithm automatically design the neural architecture. Neural Architecture Search ( NAS ) is just such a revolutionary algorithm, and the related research work is complicated and rich. Therefore, a comprehensive and systematic survey on the NAS is essential. Previously related surveys have begun to classify existing work mainly based on the key components of NAS: search space, search strategy, and evaluation strategy. While this classification method is more intuitive, it is difficult for readers to grasp the challenges and the landmark work involved. Therefore, in this survey, we provide a new perspective: beginning with an overview of the characteristics of the earliest NAS algorithms, summarizing the problems in these early NAS algorithms, and then providing solutions for subsequent related research work. In addition, we conduct a detailed and comprehensive analysis, comparison, and summary of these works. Finally, we provide some possible future research directions.

scholarly journals Neural Architecture Search for a Highly Efficient Network with Random Skip Connections

2020 ◽  
Vol 10 (11) ◽  
pp. 3712
Author(s):  
Dongjing Shan ◽  
Xiongwei Zhang ◽  
Wenhua Shi ◽  
Li Li
Keyword(s):  
State Of The Art ◽  
Cell Structure ◽  
Frequency Discrimination ◽  
Search Space ◽  
Short Term ◽  
Cell Parameters ◽  
Neural Architecture ◽  
Proposed Model ◽  
Initialization Scheme

Regarding the sequence learning of neural networks, there exists a problem of how to capture long-term dependencies and alleviate the gradient vanishing phenomenon. To manage this problem, we proposed a neural network with random connections via a scheme of a neural architecture search. First, a dense network was designed and trained to construct a search space, and then another network was generated by random sampling in the space, whose skip connections could transmit information directly over multiple periods and capture long-term dependencies more efficiently. Moreover, we devised a novel cell structure that required less memory and computational power than the structures of long short-term memories (LSTMs), and finally, we performed a special initialization scheme on the cell parameters, which could permit unhindered gradient propagation on the time axis at the beginning of training. In the experiments, we evaluated four sequential tasks: adding, copying, frequency discrimination, and image classification; we also adopted several state-of-the-art methods for comparison. The experimental results demonstrated that our proposed model achieved the best performance.

scholarly journals Exploring Neural Architecture Search Space via Deep Deterministic Sampling

IEEE Access ◽  
2021 ◽  
pp. 1-1
Author(s):  
Keith G. Mills ◽  
Mohammad Salameh ◽  
Di Niu ◽  
Fred X. Han ◽  
Seyed Saeed Changiz Rezaei ◽  
...  
Keyword(s):  
Search Space ◽  
Neural Architecture ◽  
Deterministic Sampling

scholarly journals TextNAS: A Neural Architecture Search Space Tailored for Text Representation

2020 ◽  
Vol 34 (05) ◽  
pp. 9242-9249
Author(s):  
Yujing Wang ◽  
Yaming Yang ◽  
Yiren Chen ◽  
Jing Bai ◽  
Ce Zhang ◽  
...  
Keyword(s):  
Text Classification ◽  
Network Architecture ◽  
State Of The Art ◽  
Search Space ◽  
Search Algorithms ◽  
Text Representation ◽  
Neural Architecture ◽  
Good Potential ◽  
Automatic Search ◽  
Public Datasets

Learning text representation is crucial for text classification and other language related tasks. There are a diverse set of text representation networks in the literature, and how to find the optimal one is a non-trivial problem. Recently, the emerging Neural Architecture Search (NAS) techniques have demonstrated good potential to solve the problem. Nevertheless, most of the existing works of NAS focus on the search algorithms and pay little attention to the search space. In this paper, we argue that the search space is also an important human prior to the success of NAS in different applications. Thus, we propose a novel search space tailored for text representation. Through automatic search, the discovered network architecture outperforms state-of-the-art models on various public datasets on text classification and natural language inference tasks. Furthermore, some of the design principles found in the automatic network agree well with human intuition.

scholarly journals A Method for Gradient Differentiable Network Architecture Search by Selecting and Clustering Candidate Operations

2021 ◽  
Vol 11 (23) ◽  
pp. 11436
Author(s):  
Ha Yoon Song
Keyword(s):  
Deep Learning ◽  
Network Architecture ◽  
Gradient Descent ◽  
Selection Process ◽  
Search Space ◽  
Search Cost ◽  
Second Stage ◽  
Neural Architecture ◽  
Speed Up ◽  
Mixed Operation

The current evolution of deep learning requires further optimization in terms of accuracy and time. From the perspective of new requirements, AutoML is an area that could provide possible solutions. AutoML has a neural architecture search (NAS) field. DARTS is a widely used approach in NAS and is based on gradient descent; however, it has some drawbacks. In this study, we attempted to overcome some of the drawbacks of DARTS by improving the accuracy and decreasing the search cost. The DARTS algorithm uses a mixed operation that combines all operations in the search space. The architecture parameter of each operation comprising a mixed operation is trained using gradient descent, and the operation with the largest architecture parameter is selected. The use of a mixed operation causes a problem called vote dispersion: similar operations share architecture parameters during gradient descent; thus, there are cases where the most important operation is disregarded. In this selection process, vote dispersion causes DARTS performance to degrade. To cope with this problem, we propose a new algorithm based on DARTS called DG-DARTS. Two search stages are introduced, and the clustering of operations is applied in DG-DARTS. In summary, DG-DARTS achieves an error rate of 2.51% on the CIFAR10 dataset, and its search cost is 0.2 GPU days because the search space of the second stage is reduced by half. The speed-up factor of DG-DARTS to DARTS is 6.82, which indicates that the search cost of DG-DARTS is only 13% that of DARTS.

scholarly journals A bioinspired neural architecture search based convolutional neural network for breast cancer detection using histopathology images

2021 ◽  
Vol 11 (1) ◽  
Author(s):  
Olaide N. Oyelade ◽  
Absalom E. Ezugwu
Keyword(s):  
Breast Cancer ◽  
Search Strategy ◽  
Search Algorithm ◽  
Search Space ◽  
Classification Problems ◽  
Evaluation Strategy ◽  
Neural Architecture ◽  
Histopathology Images ◽  
Block Based ◽  
Optimization Search

AbstractThe design of neural architecture to address the challenge of detecting abnormalities in histopathology images can leverage the gains made in the field of neural architecture search (NAS). The NAS model consists of a search space, search strategy and evaluation strategy. The approach supports the automation of deep learning (DL) based networks such as convolutional neural networks (CNN). Automating the process of CNN architecture engineering using this approach allows for finding the best performing network for learning classification problems in specific domains and datasets. However, the engineering process of NAS is often limited by the potential solutions in search space and the search strategy. This problem often narrows the possibility of obtaining best performing networks for challenging tasks such as the classification of breast cancer in digital histopathological samples. This study proposes a NAS model with a novel search space initialization algorithm and a new search strategy. We designed a block-based stochastic categorical-to-binary (BSCB) algorithm for generating potential CNN solutions into the search space. Also, we applied and investigated the performance of a new bioinspired optimization algorithm, namely the Ebola optimization search algorithm (EOSA), for the search strategy. The evaluation strategy was achieved through computation of loss function, architectural latency and accuracy. The results obtained using images from the BACH and BreakHis databases showed that our approach obtained best performing architectures with the top-5 of the architectures yielding a significant detection rate. The top-1 CNN architecture demonstrated a state-of-the-art performance of base on classification accuracy. The NAS strategy applied in this study and the resulting candidate architecture provides researchers with the most appropriate or suitable network configuration for using digital histopathology.

scholarly journals Generative Adversarial Neural Architecture Search

2021 ◽  
Author(s):  
Seyed Saeed Changiz Rezaei ◽  
Fred X. Han ◽  
Di Niu ◽  
Mohammad Salameh ◽  
Keith Mills ◽  
...  
Keyword(s):  
Deep Learning ◽  
Reinforcement Learning ◽  
Importance Sampling ◽  
Ad Hoc ◽  
Search Space ◽  
Learning Approach ◽  
Adversarial Learning ◽  
Search Spaces ◽  
Neural Architecture ◽  
Empirical Success

Despite the empirical success of neural architecture search (NAS) in deep learning applications, the optimality, reproducibility and cost of NAS schemes remain hard to assess. In this paper, we propose Generative Adversarial NAS (GA-NAS) with theoretically provable convergence guarantees, promoting stability and reproducibility in neural architecture search. Inspired by importance sampling, GA-NAS iteratively fits a generator to previously discovered top architectures, thus increasingly focusing on important parts of a large search space. Furthermore, we propose an efficient adversarial learning approach, where the generator is trained by reinforcement learning based on rewards provided by a discriminator, thus being able to explore the search space without evaluating a large number of architectures. Extensive experiments show that GA-NAS beats the best published results under several cases on three public NAS benchmarks. In the meantime, GA-NAS can handle ad-hoc search constraints and search spaces. We show that GA-NAS can be used to improve already optimized baselines found by other NAS methods, including EfficientNet and ProxylessNAS, in terms of ImageNet accuracy or the number of parameters, in their original search space.

Sign in / Sign up

Export Citation Format

Share Document