Design of Feedforward Neural Networks in the Classification of Hyperspectral Imagery Using Superstructural Optimization

Artificial Neural Networks (ANNs) have been used in a wide range of applications for complex datasets with their flexible mathematical architecture. The flexibility is favored by the introduction of a higher number of connections and variables, in general. However, over-parameterization of the ANN equations and the existence of redundant input variables usually result in poor test performance. This paper proposes a superstructure-based mixed-integer nonlinear programming method for optimal structural design including neuron number selection, pruning, and input selection for multilayer perceptron (MLP) ANNs. In addition, this method uses statistical measures such as the parameter covariance matrix in order to increase the test performance while permitting reduced training performance. The suggested approach was implemented on two public hyperspectral datasets (with 10% and 50% sampling ratios), namely Indian Pines and Pavia University, for the classification problem. The test results revealed promising performances compared to the standard fully connected neural networks in terms of the estimated overall and individual class accuracies. With the application of the proposed superstructural optimization, fully connected networks were pruned by over 60% in terms of the total number of connections, resulting in an increase of 4% for the 10% sampling ratio and a 1% decrease for the 50% sampling ratio. Moreover, over 20% of the spectral bands in the Indian Pines data and 30% in the Pavia University data were found statistically insignificant, and they were thus removed from the MLP networks. As a result, the proposed method was found effective in optimizing the architectural design with high generalization capabilities, particularly for fewer numbers of samples. The analysis of the eliminated spectral bands revealed that the proposed algorithm mostly removed the bands adjacent to the pre-eliminated noisy bands and highly correlated bands carrying similar information.

Download Full-text

Quantization of Deep Neural Networks for Accurate Edge Computing

ACM Journal on Emerging Technologies in Computing Systems ◽

10.1145/3451211 ◽

2021 ◽

Vol 17 (4) ◽

pp. 1-11

Author(s):

Wentao Chen ◽

Hailong Qiu ◽

Jian Zhuang ◽

Chutong Zhang ◽

Yu Hu ◽

...

Keyword(s):

Neural Network ◽

Neural Networks ◽

Image Segmentation ◽

Automatic Speech Recognition ◽

Deep Neural Networks ◽

Biomedical Image ◽

Wide Range ◽

Improve Accuracy ◽

Memory Reduction ◽

Fully Connected

Deep neural networks have demonstrated their great potential in recent years, exceeding the performance of human experts in a wide range of applications. Due to their large sizes, however, compression techniques such as weight quantization and pruning are usually applied before they can be accommodated on the edge. It is generally believed that quantization leads to performance degradation, and plenty of existing works have explored quantization strategies aiming at minimum accuracy loss. In this paper, we argue that quantization, which essentially imposes regularization on weight representations, can sometimes help to improve accuracy. We conduct comprehensive experiments on three widely used applications: fully connected network for biomedical image segmentation, convolutional neural network for image classification on ImageNet, and recurrent neural network for automatic speech recognition, and experimental results show that quantization can improve the accuracy by 1%, 1.95%, 4.23% on the three applications respectively with 3.5x-6.4x memory reduction.

Download Full-text

Deep Neural Networks Point to Mid-level Complexity of Rodent Object Vision

10.1101/2020.02.08.940189 ◽

2020 ◽

Cited By ~ 1

Author(s):

Kasper Vinken ◽

Hans Op de Beeck

Keyword(s):

Neural Networks ◽

Information Processing ◽

Object Recognition ◽

Deep Neural Networks ◽

Convincing Evidence ◽

Complex Object ◽

Visual Neuroscience ◽

Wide Range ◽

Invariant Object Recognition ◽

Fully Connected

AbstractIn the last two decades rodents have been on the rise as a dominant model for visual neuroscience. This is particularly true for earlier levels of information processing, but high-profile papers have suggested that also higher levels of processing such as invariant object recognition occur in rodents. Here we provide a quantitative and comprehensive assessment of this claim by comparing a wide range of rodent behavioral and neural data with convolutional deep neural networks. These networks have been shown to capture the richness of information processing in primates through a succession of convolutional and fully connected layers. We find that rodent object vision can be captured using low to mid-level convolutional layers only, without any convincing evidence for the need of higher layers known to simulate complex object recognition in primates. Our approach also reveals surprising insights on assumptions made before, for example, that the best performing animals would be the ones using the most complex representations – which we show to likely be incorrect. Our findings suggest a road ahead for further studies aiming at quantifying and establishing the richness of representations underlying information processing in animal models at large.

Download Full-text

An Approach of Feed-Forward Neural Network Throughput-Optimized Implementation in FPGA

Electronics ◽

10.3390/electronics9122193 ◽

2020 ◽

Vol 9 (12) ◽

pp. 2193

Author(s):

Rihards Novickis ◽

Daniels Jānis Justs ◽

Kaspars Ozols ◽

Modris Greitāns

Keyword(s):

Neural Networks ◽

Manufacturing Processes ◽

Feed Forward Neural Network ◽

Feed Forward ◽

Novel Approach ◽

Wide Range ◽

Feed Forward Neural Networks ◽

High Level ◽

Computational Resources ◽

Fully Connected

Artificial Neural Networks (ANNs) have become an accepted approach for a wide range of challenges. Meanwhile, the advancement of chip manufacturing processes is approaching saturation which calls for new computing solutions. This work presents a novel approach of an FPGA-based accelerator development for fully connected feed-forward neural networks (FFNNs). A specialized tool was developed to facilitate different implementations, which splits FFNN into elementary layers, allocates computational resources and generates high-level C++ description for high-level synthesis (HLS) tools. Various topologies are implemented and benchmarked, and a comparison with related work is provided. The proposed methodology is applied for the implementation of high-throughput virtual sensor.

Download Full-text

Data augmentation for computed tomography angiography via synthetic image generation and neural domain adaptation

Current Directions in Biomedical Engineering ◽

10.1515/cdbme-2020-0015 ◽

2020 ◽

Vol 6 (1) ◽

Author(s):

Malte Seemann ◽

Lennart Bargsten ◽

Alexander Schlaefer

Keyword(s):

Computed Tomography ◽

Neural Networks ◽

Deep Learning ◽

Medical Imaging ◽

Computed Tomography Angiography ◽

Data Augmentation ◽

Domain Adaptation ◽

Synthetic Image ◽

Wide Range ◽

The Impact

AbstractDeep learning methods produce promising results when applied to a wide range of medical imaging tasks, including segmentation of artery lumen in computed tomography angiography (CTA) data. However, to perform sufficiently, neural networks have to be trained on large amounts of high quality annotated data. In the realm of medical imaging, annotations are not only quite scarce but also often not entirely reliable. To tackle both challenges, we developed a two-step approach for generating realistic synthetic CTA data for the purpose of data augmentation. In the first step moderately realistic images are generated in a purely numerical fashion. In the second step these images are improved by applying neural domain adaptation. We evaluated the impact of synthetic data on lumen segmentation via convolutional neural networks (CNNs) by comparing resulting performances. Improvements of up to 5% in terms of Dice coefficient and 20% for Hausdorff distance represent a proof of concept that the proposed augmentation procedure can be used to enhance deep learning-based segmentation for artery lumen in CTA images.

Download Full-text

New general mixed-integer linear programming model for mobile workforce management

Optimization and Engineering ◽

10.1007/s11081-021-09597-0 ◽

2021 ◽

Author(s):

András Éles ◽

István Heckl ◽

Heriberto Cabezas

Keyword(s):

Programming Model ◽

Time Windows ◽

Mutual Exclusion ◽

Parallel Execution ◽

Mixed Integer ◽

Management Problem ◽

Routing Problem ◽

Workforce Management ◽

Computational Performance ◽

Wide Range

AbstractA mathematical model is introduced to solve a mobile workforce management problem. In such a problem there are a number of tasks to be executed at different locations by various teams. For example, when an electricity utility company has to deal with planned system upgrades and damages caused by storms. The aim is to determine the schedule of the teams in such a way that the overall cost is minimal. The mobile workforce management problem involves scheduling. The following questions should be answered: when to perform a task, how to route vehicles—the vehicle routing problem—and the order the sites should be visited and by which teams. These problems are already complex in themselves. This paper proposes an integrated mathematical programming model formulation, which, by the assignment of its binary variables, can be easily included in heuristic algorithmic frameworks. In the problem specification, a wide range of parameters can be set. This includes absolute and expected time windows for tasks, packing and unpacking in case of team movement, resource utilization, relations between tasks such as precedence, mutual exclusion or parallel execution, and team-dependent travelling and execution times and costs. To make the model able to solve larger problems, an algorithmic framework is also implemented which can be used to find heuristic solutions in acceptable time. This latter solution method can be used as an alternative. Computational performance is examined through a series of test cases in which the most important factors are scaled.

Download Full-text

High-throughput phenotyping of two plant-size traits of Eucalyptus species using neural networks

Journal of Forestry Research ◽

10.1007/s11676-021-01360-6 ◽

2021 ◽

Author(s):

Marcus Vinicius Vieira Borges ◽

Janielle de Oliveira Garcia ◽

Tays Silva Batista ◽

Alexsandra Nogueira Martins Silva ◽

Fabio Henrique Rojo Baio ◽

...

Keyword(s):

Neural Networks ◽

High Throughput ◽

Mean Squared Error ◽

Vegetation Indices ◽

Plant Size ◽

Eucalyptus Species ◽

Forest Inventories ◽

Spectral Bands ◽

High Throughput Phenotyping ◽

Input Variables

AbstractIn forest modeling to estimate the volume of wood, artificial intelligence has been shown to be quite efficient, especially using artificial neural networks (ANNs). Here we tested whether diameter at breast height (DBH) and the total plant height (Ht) of eucalyptus can be predicted at the stand level using spectral bands measured by an unmanned aerial vehicle (UAV) multispectral sensor and vegetation indices. To do so, using the data obtained by the UAV as input variables, we tested different configurations (number of hidden layers and number of neurons in each layer) of ANNs for predicting DBH and Ht at stand level for different Eucalyptus species. The experimental design was randomized blocks with four replicates, with 20 trees in each experimental plot. The treatments comprised five Eucalyptus species (E. camaldulensis, E. uroplylla, E. saligna, E. grandis, and E. urograndis) and Corymbria citriodora. DBH and Ht for each plot at the stand level were measured seven times in separate overflights by the UAV, so that the multispectral sensor could obtain spectral bands to calculate vegetation indices (VIs). ANNs were then constructed using spectral bands and VIs as input layers, in addition to the categorical variable (species), to predict DBH and Ht at the stand level simultaneously. This report represents one of the first applications of high-throughput phenotyping for plant size traits in Eucalyptus species. In general, ANNs containing three hidden layers gave better statistical performance (higher estimated r, lower estimated root mean squared error–RMSE) due to their greater capacity for self-learning. Among these ANNs, the best contained eight neurons in the first layer, seven in the second, and five in the third (8 − 7 − 5). The results reported here reveal the potential of using the generated models to perform accurate forest inventories based on spectral bands and VIs obtained with a UAV multispectral sensor and ANNs, reducing labor and time.

Download Full-text

A novel structured sparse fully connected layer in convolutional neural networks

Concurrency and Computation Practice and Experience ◽

10.1002/cpe.6213 ◽

2021 ◽

Author(s):

Naoki Matsumura ◽

Yasuaki Ito ◽

Koji Nakano ◽

Akihiko Kasagi ◽

Tsuguchika Tabaru

Keyword(s):

Neural Networks ◽

Convolutional Neural Networks ◽

Fully Connected

Download Full-text

Structural Damage Identification of Composite Rotors Based on Fully Connected Neural Networks and Convolutional Neural Networks

Sensors ◽

10.3390/s21062005 ◽

2021 ◽

Vol 21 (6) ◽

pp. 2005

Author(s):

Veronika Scholz ◽

Peter Winkler ◽

Andreas Hornig ◽

Maik Gude ◽

Angelos Filippatos

Keyword(s):

Neural Networks ◽

Convolutional Neural Networks ◽

Composite Structures ◽

Damage Identification ◽

Vibration Response ◽

Damage Initiation ◽

Matrix Cracks ◽

Operational Life ◽

Multiple Data Sets ◽

Fully Connected

Damage identification of composite structures is a major ongoing challenge for a secure operational life-cycle due to the complex, gradual damage behaviour of composite materials. Especially for composite rotors in aero-engines and wind-turbines, a cost-intensive maintenance service has to be performed in order to avoid critical failure. A major advantage of composite structures is that they are able to safely operate after damage initiation and under ongoing damage propagation. Therefore, a robust, efficient diagnostic damage identification method would allow monitoring the damage process with intervention occurring only when necessary. This study investigates the structural vibration response of composite rotors by applying machine learning methods and the ability to identify, localise and quantify the present damage. To this end, multiple fully connected neural networks and convolutional neural networks were trained on vibration response spectra from damaged composite rotors with barely visible damage, mostly matrix cracks and local delaminations using dimensionality reduction and data augmentation. A databank containing 720 simulated test cases with different damage states is used as a basis for the generation of multiple data sets. The trained models are tested using k-fold cross validation and they are evaluated based on the sensitivity, specificity and accuracy. Convolutional neural networks perform slightly better providing a performance accuracy of up to 99.3% for the damage localisation and quantification.

Download Full-text

Modeling and Optimization Sustainable Forest Supply Chain Considering Discount in Transportation System and Supplier Selection under Uncertainty

Forests ◽

10.3390/f12080964 ◽

2021 ◽

Vol 12 (8) ◽

pp. 964

Author(s):

Komeyl Baghizadeh ◽

Dominik Zimon ◽

Luay Jum’a

Keyword(s):

Supply Chain ◽

Working Hours ◽

Mixed Integer ◽

Forest Industry ◽

Northern Iran ◽

Suggested Approach ◽

Tactical Decisions ◽

Environmental Goals ◽

Forest Supply Chain ◽

The Impact

In recent decades, the forest industry has been growingly expanded due to economic conditions, climate changes, environmental and energy policies, and intense demand changes. Thus, appropriate planning is required to improve this industry. To achieve economic, social and environmental goals, a supply chain network is designed based on a multi-period and multi-product Mixed-Integer Non-Linear Programming (MINLP) model in which the objective is to maximize the profit, minimize detrimental environmental effects, improve social effects, and minimize the number of lost demands. In addition, to improve forest industry planning, strategic and tactical decisions have been implemented throughout the supply chain for all facilities, suppliers and machinery. These decisions significantly help to improve processes and product flows and to meet customers’ needs. In addition, because of the presence of uncertainty in some parameters, the proposed model was formulated and optimized under uncertainty using the hybrid robust possibilistic programming (HRPP-II) approach. The -constraint technique was used to solve the multi-objective model, and the Lagrangian relaxation (LR) method was utilized to solve the model of more complex dimensions. A case study in Northern Iran was conducted to assess the efficiency of the suggested approach. Finally, a sensitivity analysis was performed to determine the impact of important parameters on objective functions. The results of this study show that increasing the working hours of machines instead of increasing their number, increasing the capacity of some facilities instead of establishing new facilities and expanding the transport fleet has a significant impact on achieving predetermined goals.

Download Full-text