scholarly journals MergeNAS: Merge Operations into One for Differentiable Architecture Search

2020 ◽  
Author(s):  
Xiaoxing Wang ◽  
Chao Xue ◽  
Junchi Yan ◽  
Xiaokang Yang ◽  
Yonggang Hu ◽  
...  
Keyword(s):  
Mathematical Formulation ◽  
Search Space ◽  
Search Method ◽  
Test Accuracy ◽  
Search Cost ◽  
Neural Architecture ◽  
Different Types ◽  
Search Approach ◽  
Memory Utilization

Differentiable architecture search (DARTS) has been a promising one-shot architecture search approach for its mathematical formulation and competitive results. However, besides its caused high memory utilization and a large computation requirement, many research works have shown that DARTS also often suffers notable over-fitting and thus does not work robustly for some new tasks. In this paper, we propose a one-shot neural architecture search method referred to as MergeNAS by merging different types of operations e.g. convolutions into one operation. This merge-based approach not only reduces the search cost (about half a GPU day), but also alleviates over-fitting by reducing the redundant parameters. Extensive experiments on different search space and various datasets have been conducted to verify our approach, showing that MergeNAS can converge to a stable architecture and achieve better performance with fewer parameters and search cost. For test accuracy and its stability, MergeNAS outperforms all NAS baseline methods implemented on NAS-Bench-201, including DARTS, ENAS, RS, BOHB, GDAS and hand-crafted ResNet.

scholarly journals A Method for Gradient Differentiable Network Architecture Search by Selecting and Clustering Candidate Operations

2021 ◽  
Vol 11 (23) ◽  
pp. 11436
Author(s):  
Ha Yoon Song
Keyword(s):  
Deep Learning ◽  
Network Architecture ◽  
Gradient Descent ◽  
Selection Process ◽  
Search Space ◽  
Search Cost ◽  
Second Stage ◽  
Neural Architecture ◽  
Speed Up ◽  
Mixed Operation

The current evolution of deep learning requires further optimization in terms of accuracy and time. From the perspective of new requirements, AutoML is an area that could provide possible solutions. AutoML has a neural architecture search (NAS) field. DARTS is a widely used approach in NAS and is based on gradient descent; however, it has some drawbacks. In this study, we attempted to overcome some of the drawbacks of DARTS by improving the accuracy and decreasing the search cost. The DARTS algorithm uses a mixed operation that combines all operations in the search space. The architecture parameter of each operation comprising a mixed operation is trained using gradient descent, and the operation with the largest architecture parameter is selected. The use of a mixed operation causes a problem called vote dispersion: similar operations share architecture parameters during gradient descent; thus, there are cases where the most important operation is disregarded. In this selection process, vote dispersion causes DARTS performance to degrade. To cope with this problem, we propose a new algorithm based on DARTS called DG-DARTS. Two search stages are introduced, and the clustering of operations is applied in DG-DARTS. In summary, DG-DARTS achieves an error rate of 2.51% on the CIFAR10 dataset, and its search cost is 0.2 GPU days because the search space of the second stage is reduced by half. The speed-up factor of DG-DARTS to DARTS is 6.82, which indicates that the search cost of DG-DARTS is only 13% that of DARTS.

scholarly journals Neural Inheritance Relation Guided One-Shot Layer Assignment Search

2020 ◽  
Vol 34 (04) ◽  
pp. 5158-5165
Author(s):  
Rang Meng ◽  
Weijie Chen ◽  
Di Xie ◽  
Yuan Zhang ◽  
Shiliang Pu
Keyword(s):  
Network Performance ◽  
Search Space ◽  
Independent Research ◽  
Neural Architecture ◽  
Network Search ◽  
Layer Assignment ◽  
Search Approach ◽  
The Impact ◽  
First Time ◽  
Strongly Consistent

Layer assignment is seldom picked out as an independent research topic in neural architecture search. In this paper, for the first time, we systematically investigate the impact of different layer assignments to the network performance by building an architecture dataset of layer assignment on CIFAR-100. Through analyzing this dataset, we discover a neural inheritance relation among the networks with different layer assignments, that is, the optimal layer assignments for deeper networks always inherit from those for shallow networks. Inspired by this neural inheritance relation, we propose an efficient one-shot layer assignment search approach via inherited sampling. Specifically, the optimal layer assignment searched in the shallow network can be provided as a strong sampling priori to train and search the deeper ones in supernet, which extremely reduces the network search space. Comprehensive experiments carried out on CIFAR-100 illustrate the efficiency of our proposed method. Our search results are strongly consistent with the optimal ones directly selected from the architecture dataset. To further confirm the generalization of our proposed method, we also conduct experiments on Tiny-ImageNet and ImageNet. Our searched results are remarkably superior to the handcrafted ones under the unchanged computational budgets. The neural inheritance relation discovered in this paper can provide insights to the universal neural architecture search.

scholarly journals AutoShrink: A Topology-Aware NAS for Discovering Efficient Neural Architecture

2020 ◽  
Vol 34 (04) ◽  
pp. 6829-6836
Author(s):  
Tunhou Zhang ◽  
Hsin-Pai Cheng ◽  
Zhenwen Li ◽  
Feng Yan ◽  
Chengyu Huang ◽  
...  
Keyword(s):  
Cell Structure ◽  
Search Time ◽  
Building Blocks ◽  
Search Space ◽  
Directed Acyclic Graphs ◽  
Neural Architecture ◽  
Cell Structures ◽  
Acyclic Graphs ◽  
Search Approach ◽  
Network Patterns

Resource is an important constraint when deploying Deep Neural Networks (DNNs) on mobile and edge devices. Existing works commonly adopt the cell-based search approach, which limits the flexibility of network patterns in learned cell structures. Moreover, due to the topology-agnostic nature of existing works, including both cell-based and node-based approaches, the search process is time consuming and the performance of found architecture may be sub-optimal. To address these problems, we propose AutoShrink, a topology-aware Neural Architecture Search (NAS) for searching efficient building blocks of neural architectures. Our method is node-based and thus can learn flexible network patterns in cell structures within a topological search space. Directed Acyclic Graphs (DAGs) are used to abstract DNN architectures and progressively optimize the cell structure through edge shrinking. As the search space intrinsically reduces as the edges are progressively shrunk, AutoShrink explores more flexible search space with even less search time. We evaluate AutoShrink on image classification and language tasks by crafting ShrinkCNN and ShrinkRNN models. ShrinkCNN is able to achieve up to 48% parameter reduction and save 34% Multiply-Accumulates (MACs) on ImageNet-1K with comparable accuracy of state-of-the-art (SOTA) models. Specifically, both ShrinkCNN and ShrinkRNN are crafted within 1.5 GPU hours, which is 7.2× and 6.7× faster than the crafting time of SOTA CNN and RNN models, respectively.

Automatic Mining of Quantitative Association Rules with Gravitational Search Algorithm

2017 ◽  
Vol 27 (03) ◽  
pp. 343-372 ◽  
Author(s):  
Umit Can ◽  
Bilal Alatas
Keyword(s):  
Association Rules ◽  
Optimization Problems ◽  
Search Algorithm ◽  
Fitness Function ◽  
Optimization Algorithms ◽  
Gravitational Search Algorithm ◽  
Search Space ◽  
Search Method ◽  
Quantitative Association Rules ◽  
Gravitational Search

The classical optimization algorithms are not efficient in solving complex search and optimization problems. Thus, some heuristic optimization algorithms have been proposed. In this paper, exploration of association rules within numerical databases with Gravitational Search Algorithm (GSA) has been firstly performed. GSA has been designed as search method for quantitative association rules from the databases which can be regarded as search space. Furthermore, determining the minimum values of confidence and support for every database which is a hard job has been eliminated by GSA. Apart from this, the fitness function used for GSA is very flexible. According to the interested problem, some parameters can be removed from or added to the fitness function. The range values of the attributes have been automatically adjusted during the time of mining of the rules. That is why there is not any requirements for the pre-processing of the data. Attributes interaction problem has also been eliminated with the designed GSA. GSA has been tested with four real databases and promising results have been obtained. GSA seems an effective search method for complex numerical sequential patterns mining, numerical classification rules mining, and clustering rules mining tasks of data mining.

A Comprehensive Survey of Neural Architecture Search

2021 ◽  
Vol 54 (4) ◽  
pp. 1-34
Author(s):  
Pengzhen Ren ◽  
Yun Xiao ◽  
Xiaojun Chang ◽  
Po-yao Huang ◽  
Zhihui Li ◽  
...  
Keyword(s):  
Research Work ◽  
Search Space ◽  
Feature Representation ◽  
Future Research ◽  
Related Research ◽  
Neural Architecture ◽  
Comprehensive Survey ◽  
Future Research Directions ◽  
Intuitive Idea ◽  
New Perspective

Deep learning has made substantial breakthroughs in many fields due to its powerful automatic representation capabilities. It has been proven that neural architecture design is crucial to the feature representation of data and the final performance. However, the design of the neural architecture heavily relies on the researchers’ prior knowledge and experience. And due to the limitations of humans’ inherent knowledge, it is difficult for people to jump out of their original thinking paradigm and design an optimal model. Therefore, an intuitive idea would be to reduce human intervention as much as possible and let the algorithm automatically design the neural architecture. Neural Architecture Search ( NAS ) is just such a revolutionary algorithm, and the related research work is complicated and rich. Therefore, a comprehensive and systematic survey on the NAS is essential. Previously related surveys have begun to classify existing work mainly based on the key components of NAS: search space, search strategy, and evaluation strategy. While this classification method is more intuitive, it is difficult for readers to grasp the challenges and the landmark work involved. Therefore, in this survey, we provide a new perspective: beginning with an overview of the characteristics of the earliest NAS algorithms, summarizing the problems in these early NAS algorithms, and then providing solutions for subsequent related research work. In addition, we conduct a detailed and comprehensive analysis, comparison, and summary of these works. Finally, we provide some possible future research directions.

Minimization of total trim loss occurring in pipe cutting in shipbuilding with genetic algorithm

2021 ◽  
pp. 147509022199169
Author(s):  
Abdullah Türk ◽  
Dursun Saral ◽  
Murat Özkök ◽  
Ercan Köse
Keyword(s):  
Genetic Algorithm ◽  
Genetic Algorithms ◽  
Search Space ◽  
Cutting Process ◽  
Genetic Operators ◽  
Critical Stage ◽  
Different Types ◽  
Trim Loss ◽  
Pipe Systems

Outfitting is a critical stage in the shipbuilding process. Within the outfitting, the construction of pipe systems is a phase that has a significant effect on time and cost. While cutting the pipes required for the pipe systems in shipyards, the cutting process is usually performed randomly. This can result in large amounts of trim losses. In this paper, we present an approach to minimize these losses. With the proposed method it is aimed to base the pipe cutting process on a specific systematic. To solve this problem, Genetic Algorithms (GA), which gives successful results in solving many problems in the literature, have been used. Different types of genetic operators have been used to investigate the search space of the problem well. The results obtained have proven the effectiveness of the proposed approach.

CellNet: An Improved Neural Architecture Search Method for Coal and Gangue Classification

2021 ◽  
Author(s):  
Lirong Sun ◽  
Wenbo Zhu ◽  
Qinghua Lu ◽  
Aiyuan Li ◽  
Lufeng Luo ◽  
...  
Keyword(s):  
Search Method ◽  
Neural Architecture

Data Clustering Using Sine Cosine Algorithm

2017 ◽  
pp. 715-726 ◽  
Author(s):  
Vijay Kumar ◽  
Dinesh Kumar
Keyword(s):  
Real Life ◽  
Quality Measures ◽  
Search Space ◽  
Search Method ◽  
Local Optima ◽  
Encoding Scheme ◽  
Clustering Techniques ◽  
Sine Cosine Algorithm ◽  
Cluster Quality ◽  
Optimal Cluster

The clustering techniques suffer from cluster centers initialization and local optima problems. In this chapter, the new metaheuristic algorithm, Sine Cosine Algorithm (SCA), is used as a search method to solve these problems. The SCA explores the search space of given dataset to find out the near-optimal cluster centers. The center based encoding scheme is used to evolve the cluster centers. The proposed SCA-based clustering technique is evaluated on four real-life datasets. The performance of SCA-based clustering is compared with recently developed clustering techniques. The experimental results reveal that SCA-based clustering gives better values in terms of cluster quality measures.

scholarly journals Study of the measurements placement problem for planning of the optimal experiment

2019 ◽  
Vol 102 ◽  
pp. 03006
Author(s):  
Oksana A. Grebneva
Keyword(s):  
Finite Number ◽  
Global Solution ◽  
Current Literature ◽  
Mathematical Formulation ◽  
Optimal Placement ◽  
Placement Problem ◽  
Optimal Experiment ◽  
Measuring Devices ◽  
Different Types ◽  
Pipeline Systems

Both in Russia and abroad, there are works that are devoted to the problem of optimal placement of measuring devices, which are evidenced by the current literature. The proposed methods are not universal, that does not allow them to be directly used for different types of pipeline systems. In addition, the developed algorithms does not guarantee a global solution. In this regard, there is a demand for solving the problem of optimal placement of measuring devices for pipeline systems. At the same time, not only the number and accuracy of measuring devices, but also their composition and placement locations are important. In this paper, a mathematical formulation of the problem of optimal placement of measuring devices is given, methods for its solution are proposed. The numerical example shows the effectiveness of the proposed method of optimal placement of measuring devices, which allows to get a global solution for a previously known finite number of steps.

Sign in / Sign up

Export Citation Format

Share Document