scholarly journals Neural Graph Embedding for Neural Architecture Search

2020 ◽  
Vol 34 (04) ◽  
pp. 4707-4714 ◽  
Author(s):  
Wei Li ◽  
Shaogang Gong ◽  
Xiatian Zhu
Keyword(s):  
Neural Networks ◽  
Graph Embedding ◽  
Semantic Segmentation ◽  
Directed Acyclic Graphs ◽  
Network Architectures ◽  
Search Performance ◽  
Convolutional Network ◽  
Neural Architecture ◽  
Acyclic Graphs ◽  
Topology Information

Existing neural architecture search (NAS) methods often operate in discrete or continuous spaces directly, which ignores the graphical topology knowledge of neural networks. This leads to suboptimal search performance and efficiency, given the factor that neural networks are essentially directed acyclic graphs (DAG). In this work, we address this limitation by introducing a novel idea of neural graph embedding (NGE). Specifically, we represent the building block (i.e. the cell) of neural networks with a neural DAG, and learn it by leveraging a Graph Convolutional Network to propagate and model the intrinsic topology information of network architectures. This results in a generic neural network representation integrable with different existing NAS frameworks. Extensive experiments show the superiority of NGE over the state-of-the-art methods on image classification and semantic segmentation.

scholarly journals TGV Upsampling: A Making-Up Operation for Semantic Segmentation

2019 ◽  
Vol 2019 ◽  
pp. 1-12
Author(s):  
Xu Yin ◽  
Yan Li ◽  
Byeong-Seok Shin
Keyword(s):  
Neural Networks ◽  
Deep Learning ◽  
Detailed Analysis ◽  
Semantic Segmentation ◽  
Information Loss ◽  
Network Architectures ◽  
Feature Maps ◽  
Extraction Processes ◽  
Multiple Uses ◽  
Improved Accuracy

With the widespread use of deep learning methods, semantic segmentation has achieved great improvements in recent years. However, many researchers have pointed out that with multiple uses of convolution and pooling operations, great information loss would occur in the extraction processes. To solve this problem, various operations or network architectures have been suggested to make up for the loss of information. We observed a trend in many studies to design a network as a symmetric type, with both parts representing the “encoding” and “decoding” stages. By “upsampling” operations in the “decoding” stage, feature maps are constructed in a certain way that would more or less make up for the losses in previous layers. In this paper, we focus on upsampling operations, make a detailed analysis, and compare current methods used in several famous neural networks. We also combine the knowledge on image restoration and design a new upsampled layer (or operation) named the TGV upsampling algorithm. We successfully replaced upsampling layers in the previous research with our new method. We found that our model can better preserve detailed textures and edges of feature maps and can, on average, achieve 1.4–2.3% improved accuracy compared to the original models.

scholarly journals A method of searching and marking artifacts in images applying detection and segmentation algorithms

2021 ◽  
pp. 7-18
Author(s):  
Andrey M. Kitenko ◽  
Keyword(s):  
Neural Networks ◽  
Network Models ◽  
Semantic Segmentation ◽  
Document Processing ◽  
Neural Network Models ◽  
Convolutional Network ◽  
Human Errors ◽  
Search And Selection ◽  
Stress Fatigue ◽  
Selection Of

The paper explores the possibility of using neural networks to single out target artifacts on different types of documents. Numerous types of neural networks are often used for document processing, from text analysis to the allocation of certain areas where the desired information may be contained. However, to date, there are no perfect document processing systems that can work autonomously, compensating for human errors that may appear in the process of work due to stress, fatigue and many other reasons. In this work, the emphasis is on the search and selection of target artifacts in drawings, in conditions of a small amount of initial data. The proposed method of searching and highlighting artifacts in the image consists of two main parts, detection and semantic segmentation of the detected area. The method is based on training with a teacher on marked-up data for two convolutional neural networks. The first convolutional network is used to detect an area with an artifact, in this example YoloV4 was taken as the basis. For semantic segmentation, the U-Net architecture is used, where the basis is the pre-trained Efficientnetb0. By combining these neural networks, good results were achieved, even for the selection of certain handwritten texts, without using the specifics of building neural network models for text recognition. This method can be used to search for and highlight artifacts in large datasets, while the artifacts themselves may be different in shape, color and type, and they may be located in different places of the image, have or not have intersection with other objects.

scholarly journals Dual Self-Paced Graph Convolutional Network: Towards Reducing Attribute Distortions Induced by Topology

2019 ◽  
Author(s):  
Liang Yang ◽  
Zhiyang Chen ◽  
Junhua Gu ◽  
Yuanfang Guo
Keyword(s):  
Neural Networks ◽  
Learning Strategies ◽  
Convolutional Neural Networks ◽  
Training Process ◽  
Training Set ◽  
Convolutional Network ◽  
Attribute Information ◽  
Topology Information ◽  
Node Classification ◽  
The Impact

The success of graph convolutional neural networks (GCNNs) based semi-supervised node classification is credited to the attribute smoothing (propagating) over the topology. However, the attributes may be interfered by the utilization of the topology information. This distortion will induce a certain amount of misclassifications of the nodes, which can be correctly predicted with only the attributes. By analyzing the impact of the edges in attribute propagations, the simple edges, which connect two nodes with similar attributes, should be given priority during the training process compared to the complex ones according to curriculum learning. To reduce the distortions induced by the topology while exploit more potentials of the attribute information, Dual Self-Paced Graph Convolutional Network (DSP-GCN) is proposed in this paper. Specifically, the unlabelled nodes with confidently predicted labels are gradually added into the training set in the node-level self-paced learning, while edges are gradually, from the simple edges to the complex ones, added into the graph during the training process in the edge-level self-paced learning. These two learning strategies are designed to mutually reinforce each other by coupling the selections of the edges and unlabelled nodes. Experimental results of transductive semi-supervised node classification on many real networks indicate that the proposed DSP-GCN has successfully reduced the attribute distortions induced by the topology while it gives superior performances with only one graph convolutional layer.

scholarly journals Scaling All-Goals Updates in Reinforcement Learning Using Convolutional Neural Networks

2020 ◽  
Vol 34 (04) ◽  
pp. 5355-5362
Author(s):  
Fabio Pardo ◽  
Vitaly Levdik ◽  
Petar Kormushev
Keyword(s):  
Neural Networks ◽  
Reinforcement Learning ◽  
Convolutional Neural Networks ◽  
Network Architectures ◽  
Convolutional Network ◽  
Agent Learning ◽  
Q Values

Being able to reach any desired location in the environment can be a valuable asset for an agent. Learning a policy to navigate between all pairs of states individually is often not feasible. An all-goals updating algorithm uses each transition to learn Q-values towards all goals simultaneously and off-policy. However the expensive numerous updates in parallel limited the approach to small tabular cases so far. To tackle this problem we propose to use convolutional network architectures to generate Q-values and updates for a large number of goals at once. We demonstrate the accuracy and generalization qualities of the proposed method on randomly generated mazes and Sokoban puzzles. In the case of on-screen goal coordinates the resulting mapping from frames to distance-maps directly informs the agent about which places are reachable and in how many steps. As an example of application we show that replacing the random actions in ε-greedy exploration by several actions towards feasible goals generates better exploratory trajectories on Montezuma's Revenge and Super Mario All-Stars games.

scholarly journals DropConnect is effective in modeling uncertainty of Bayesian deep networks

2021 ◽  
Vol 11 (1) ◽  
Author(s):  
Aryan Mobiny ◽  
Pengyu Yuan ◽  
Supratik K. Moulik ◽  
Naveen Garg ◽  
Carol C. Wu ◽  
...  
Keyword(s):  
Neural Networks ◽  
Prediction Accuracy ◽  
State Of The Art ◽  
Computational Cost ◽  
Semantic Segmentation ◽  
Autonomous Driving ◽  
Learning Technology ◽  
Network Architectures ◽  
Uncertainty Estimates ◽  
Deep Networks

AbstractDeep neural networks (DNNs) have achieved state-of-the-art performance in many important domains, including medical diagnosis, security, and autonomous driving. In domains where safety is highly critical, an erroneous decision can result in serious consequences. While a perfect prediction accuracy is not always achievable, recent work on Bayesian deep networks shows that it is possible to know when DNNs are more likely to make mistakes. Knowing what DNNs do not know is desirable to increase the safety of deep learning technology in sensitive applications; Bayesian neural networks attempt to address this challenge. Traditional approaches are computationally intractable and do not scale well to large, complex neural network architectures. In this paper, we develop a theoretical framework to approximate Bayesian inference for DNNs by imposing a Bernoulli distribution on the model weights. This method called Monte Carlo DropConnect (MC-DropConnect) gives us a tool to represent the model uncertainty with little change in the overall model structure or computational cost. We extensively validate the proposed algorithm on multiple network architectures and datasets for classification and semantic segmentation tasks. We also propose new metrics to quantify uncertainty estimates. This enables an objective comparison between MC-DropConnect and prior approaches. Our empirical results demonstrate that the proposed framework yields significant improvement in both prediction accuracy and uncertainty estimation quality compared to the state of the art.

scholarly journals Binarized Neural Architecture Search

2020 ◽  
Vol 34 (07) ◽  
pp. 10526-10533 ◽  
Author(s):  
Hanlin Chen ◽  
Li'an Zhuo ◽  
Baochang Zhang ◽  
Xiawu Zheng ◽  
Jianzhuang Liu ◽  
...  
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
State Of The Art ◽  
Optimization Methods ◽  
Search Space ◽  
Network Architectures ◽  
Neural Architecture ◽  
Space Reduction ◽  
The Cost ◽  
A Performance

Neural architecture search (NAS) can have a significant impact in computer vision by automatically designing optimal neural network architectures for various tasks. A variant, binarized neural architecture search (BNAS), with a search space of binarized convolutions, can produce extremely compressed models. Unfortunately, this area remains largely unexplored. BNAS is more challenging than NAS due to the learning inefficiency caused by optimization requirements and the huge architecture space. To address these issues, we introduce channel sampling and operation space reduction into a differentiable NAS to significantly reduce the cost of searching. This is accomplished through a performance-based strategy used to abandon less potential operations. Two optimization methods for binarized neural networks are used to validate the effectiveness of our BNAS. Extensive experiments demonstrate that the proposed BNAS achieves a performance comparable to NAS on both CIFAR and ImageNet databases. An accuracy of 96.53% vs. 97.22% is achieved on the CIFAR-10 dataset, but with a significantly compressed model, and a 40% faster search than the state-of-the-art PC-DARTS.

scholarly journals Robustness Verification of Semantic Segmentation Neural Networks Using Relaxed Reachability

2021 ◽  
pp. 263-286
Author(s):  
Hoang-Dung Tran ◽  
Neelanjana Pal ◽  
Patrick Musau ◽  
Diego Manzanas Lopez ◽  
Nathaniel Hamilton ◽  
...  
Keyword(s):  
Neural Networks ◽  
Robustness Analysis ◽  
Semantic Segmentation ◽  
Network Architectures ◽  
Verification Methods ◽  
Verification Time ◽  
Linear Programming Problems ◽  
Total Analysis ◽  
Evaluation Measure ◽  
Robust Variant

AbstractThis paper introduces robustness verification for semantic segmentation neural networks (in short, semantic segmentation networks [SSNs]), building on and extending recent approaches for robustness verification of image classification neural networks. Despite recent progress in developing verification methods for specifications such as local adversarial robustness in deep neural networks (DNNs) in terms of scalability, precision, and applicability to different network architectures, layers, and activation functions, robustness verification of semantic segmentation has not yet been considered. We address this limitation by developing and applying new robustness analysis methods for several segmentation neural network architectures, specifically by addressing reachability analysis of up-sampling layers, such as transposed convolution and dilated convolution. We consider several definitions of robustness for segmentation, such as the percentage of pixels in the output that can be proven robust under different adversarial perturbations, and a robust variant of intersection-over-union (IoU), the typical performance evaluation measure for segmentation tasks. Our approach is based on a new relaxed reachability method, allowing users to select the percentage of a number of linear programming problems (LPs) to solve when constructing the reachable set, through a relaxation factor percentage. The approach is implemented within NNV, then applied and evaluated on segmentation datasets, such as a multi-digit variant of MNIST known as M2NIST. Thorough experiments show that by using transposed convolution for up-sampling and average-pooling for down-sampling, combined with minimizing the number of ReLU layers in the SSNs, we can obtain SSNs with not only high accuracy (IoU), but also that are more robust to adversarial attacks and amenable to verification. Additionally, using our new relaxed reachability method, we can significantly reduce the verification time for neural networks whose ReLU layers dominate the total analysis time, even in classification tasks.

scholarly journals A Genetic Programming Approach to Designing Convolutional Neural Network Architectures

2018 ◽  
Author(s):  
Masanori Suganuma ◽  
Shinichi Shirakawa ◽  
Tomoharu Nagao
Keyword(s):  
Neural Network ◽  
Genetic Programming ◽  
Convolutional Neural Network ◽  
State Of The Art ◽  
Directed Acyclic Graphs ◽  
Validation Dataset ◽  
Programming Approach ◽  
Network Architectures ◽  
Cartesian Genetic Programming ◽  
Acyclic Graphs

We propose a method for designing convolutional neural network (CNN) architectures based on Cartesian genetic programming (CGP). In the proposed method, the architectures of CNNs are represented by directed acyclic graphs, in which each node represents highly-functional modules such as convolutional blocks and tensor operations, and each edge represents the connectivity of layers. The architecture is optimized to maximize the classification accuracy for a validation dataset by an evolutionary algorithm. We show that the proposed method can find competitive CNN architectures compared with state-of-the-art methods on the image classification task using CIFAR-10 and CIFAR-100 datasets.

DeepGCNs-Att: Point cloud semantic segmentation with contextual point representations

2021 ◽  
pp. 1-10
Author(s):  
Bin Jiang ◽  
Xinyu Wang ◽  
Li Huang ◽  
Jian Xiao
Keyword(s):  
Neural Networks ◽  
Point Cloud ◽  
Semantic Segmentation ◽  
Global Features ◽  
Convolutional Network ◽  
Convolutional Networks ◽  
Network Layers ◽  
Speed Up ◽  
Network Backbone ◽  
Context Features

 Graph Convolutional Networks are able to characterize non-Euclidean spaces effectively compared with traditional Convolutional Neural Networks, which can extract the local features of the point cloud using deep neural networks, but it cannot make full use of the global features of the point cloud for semantic segmentation. To solve this problem, this paper proposes a novel network structure called DeepGCNs-Att that enables deep Graph Convolutional Network to aggregate global context features efficiently. Moreover, to speed up the computation, we add an Attention layer after the Graph Convolutional Network Backbone Block to mutually enhance the connection between the distant points of the non-Euclidean space. Our model is tested on the standard benchmark S3DIS. By comparing with other deep Graph Convolutional Networks, our DeepGCNs-Att’s mIoU has at least two percent higher than that of all other models and even shows excellent results in space complexity and computational complexity under the same number of Graph Convolutional Network layers.

scholarly journals AutoShrink: A Topology-Aware NAS for Discovering Efficient Neural Architecture

2020 ◽  
Vol 34 (04) ◽  
pp. 6829-6836
Author(s):  
Tunhou Zhang ◽  
Hsin-Pai Cheng ◽  
Zhenwen Li ◽  
Feng Yan ◽  
Chengyu Huang ◽  
...  
Keyword(s):  
Cell Structure ◽  
Search Time ◽  
Building Blocks ◽  
Search Space ◽  
Directed Acyclic Graphs ◽  
Neural Architecture ◽  
Cell Structures ◽  
Acyclic Graphs ◽  
Search Approach ◽  
Network Patterns

Resource is an important constraint when deploying Deep Neural Networks (DNNs) on mobile and edge devices. Existing works commonly adopt the cell-based search approach, which limits the flexibility of network patterns in learned cell structures. Moreover, due to the topology-agnostic nature of existing works, including both cell-based and node-based approaches, the search process is time consuming and the performance of found architecture may be sub-optimal. To address these problems, we propose AutoShrink, a topology-aware Neural Architecture Search (NAS) for searching efficient building blocks of neural architectures. Our method is node-based and thus can learn flexible network patterns in cell structures within a topological search space. Directed Acyclic Graphs (DAGs) are used to abstract DNN architectures and progressively optimize the cell structure through edge shrinking. As the search space intrinsically reduces as the edges are progressively shrunk, AutoShrink explores more flexible search space with even less search time. We evaluate AutoShrink on image classification and language tasks by crafting ShrinkCNN and ShrinkRNN models. ShrinkCNN is able to achieve up to 48% parameter reduction and save 34% Multiply-Accumulates (MACs) on ImageNet-1K with comparable accuracy of state-of-the-art (SOTA) models. Specifically, both ShrinkCNN and ShrinkRNN are crafted within 1.5 GPU hours, which is 7.2× and 6.7× faster than the crafting time of SOTA CNN and RNN models, respectively.

Sign in / Sign up

Export Citation Format

Share Document