Fusion neural networks for plant classification: learning to combine RGB, hyperspectral, and lidar data

Airborne remote sensing offers unprecedented opportunities to efficiently monitor vegetation, but methods to delineate and classify individual plant species using the collected data are still actively being developed and improved. The Integrating Data science with Trees and Remote Sensing (IDTReeS) plant identification competition openly invited scientists to create and compare individual tree mapping methods. Participants were tasked with training taxon identification algorithms based on two sites, to then transfer their methods to a third unseen site, using field-based plant observations in combination with airborne remote sensing image data products from the National Ecological Observatory Network (NEON). These data were captured by a high resolution digital camera sensitive to red, green, blue (RGB) light, hyperspectral imaging spectrometer spanning the visible to shortwave infrared wavelengths, and lidar systems to capture the spectral and structural properties of vegetation. As participants in the IDTReeS competition, we developed a two-stage deep learning approach to integrate NEON remote sensing data from all three sensors and classify individual plant species and genera. The first stage was a convolutional neural network that generates taxon probabilities from RGB images, and the second stage was a fusion neural network that “learns” how to combine these probabilities with hyperspectral and lidar data. Our two-stage approach leverages the ability of neural networks to flexibly and automatically extract descriptive features from complex image data with high dimensionality. Our method achieved an overall classification accuracy of 0.51 based on the training set, and 0.32 based on the test set which contained data from an unseen site with unknown taxa classes. Although transferability of classification algorithms to unseen sites with unknown species and genus classes proved to be a challenging task, developing methods with openly available NEON data that will be collected in a standardized format for 30 years allows for continual improvements and major gains for members of the computational ecology community. We outline promising directions related to data preparation and processing techniques for further investigation, and provide our code to contribute to open reproducible science efforts.

Download Full-text

WaterNet: A Convolutional Neural Network for Chlorophyll-a Concentration Retrieval

Remote Sensing ◽

10.3390/rs12121966 ◽

2020 ◽

Vol 12 (12) ◽

pp. 1966 ◽

Cited By ~ 2

Author(s):

Muhammad Aldila Syariz ◽

Chao-Hung Lin ◽

Manh Van Nguyen ◽

Lalu Muhamad Jaelani ◽

Ariel C. Blanco

Keyword(s):

Neural Network ◽

Remote Sensing ◽

Neural Networks ◽

Convolutional Neural Network ◽

Chlorophyll A ◽

Remote Sensing Images ◽

Two Stage ◽

Chl A ◽

The Neural Network

The retrieval of chlorophyll-a (Chl-a) concentrations relies on empirical or analytical analyses, which generally experience difficulties from the diversity of inland waters in statistical analyses and the complexity of radiative transfer equations in analytical analyses, respectively. Previous studies proposed the utilization of artificial neural networks (ANNs) to alleviate these problems. However, ANNs do not consider the problem of insufficient in situ samples during model training, and they do not fully utilize the spatial and spectral information of remote sensing images in neural networks. In this study, a two-stage training is introduced to address the problem regarding sample insufficiency. The neural network is pretrained using the samples derived from an existing Chl-a concentration model in the first stage, and the pretrained model is refined with in situ samples in the second stage. A novel convolutional neural network for Chl-a concentration retrieval called WaterNet is proposed which utilizes both spectral and spatial information of remote sensing images. In addition, an end-to-end structure that integrates feature extraction, band expansion, and Chl-a estimation into the neural network leads to an efficient and effective Chl-a concentration retrieval. In experiments, Sentinel-3 images with the same acquisition days of in situ measurements over Laguna Lake in the Philippines were used to train and evaluate WaterNet. The quantitative analyses show that the two-stage training is more likely than the one-stage training to reach the global optimum in the optimization, and WaterNet with two-stage training outperforms, in terms of estimation accuracy, related ANN-based and band-combination-based Chl-a concentration models.

Download Full-text

LAND COVER CLASSIFICATION BASED ON MODIS IMAGERY DATA USING ARTIFICIAL NEURAL NETWORKS

Environment Technology Resources Proceedings of the International Scientific and Practical Conference ◽

10.17770/etr2017vol2.2545 ◽

2017 ◽

Vol 2 ◽

pp. 159

Author(s):

Arthur Stepchenko

Keyword(s):

Neural Network ◽

Remote Sensing ◽

Neural Networks ◽

Artificial Neural Network ◽

Artificial Neural Networks ◽

Land Cover ◽

Classification System ◽

Image Data ◽

Automated Classification ◽

Artificial Neural

Remote sensing has been widely used to obtain land cover information using automated classification. Land cover is a measure of what is overlaying the surface of the earth. Accurate mapping of land cover on a regional scale is useful in such fields as precision agriculture or forest management and is one of the most important applications in remote sensing. In this study, multispectral MODIS Terra NDVI images and an artificial neural network (ANN) were used in land cover classification. Artificial neural network is a computing tool that is designed to simulate the way the human brain analyzes and process information. Artificial neural networks are one of the commonly applied machine learning algorithm, and they have become popular in the analysis of remotely sensed data, particularly in classification or feature extraction from image data more accurately than conventional method. This paper focuses on an automated classification system based on a pattern recognition neural network. Variational mode decomposition method is used as an image data pre-processing tool in this classification system. The result of this study will be land cover map.

Download Full-text

Convolutional Neural Network for the Semantic Segmentation of Remote Sensing Images

Mobile Networks and Applications ◽

10.1007/s11036-020-01703-3 ◽

2021 ◽

Vol 26 (1) ◽

pp. 200-215

Author(s):

Muhammad Alam ◽

Jian-Feng Wang ◽

Cong Guangpei ◽

LV Yunrong ◽

Yuanfang Chen

Keyword(s):

Neural Network ◽

Remote Sensing ◽

Neural Networks ◽

Image Processing ◽

Deep Learning ◽

Semantic Segmentation ◽

Natural Scene ◽

Remote Sensing Images ◽

Advantages And Disadvantages ◽

Target Segmentation

AbstractIn recent years, the success of deep learning in natural scene image processing boosted its application in the analysis of remote sensing images. In this paper, we applied Convolutional Neural Networks (CNN) on the semantic segmentation of remote sensing images. We improve the Encoder- Decoder CNN structure SegNet with index pooling and U-net to make them suitable for multi-targets semantic segmentation of remote sensing images. The results show that these two models have their own advantages and disadvantages on the segmentation of different objects. In addition, we propose an integrated algorithm that integrates these two models. Experimental results show that the presented integrated algorithm can exploite the advantages of both the models for multi-target segmentation and achieve a better segmentation compared to these two models.

Download Full-text

Road Extraction from Unmanned Aerial Vehicle Remote Sensing Images Based on Improved Neural Networks

Sensors ◽

10.3390/s19194115 ◽

2019 ◽

Vol 19 (19) ◽

pp. 4115 ◽

Cited By ~ 1

Author(s):

Yuxia Li ◽

Bo Peng ◽

Lei He ◽

Kunlong Fan ◽

Zhenxu Li ◽

...

Keyword(s):

Neural Network ◽

Remote Sensing ◽

Neural Networks ◽

Unmanned Aerial Vehicle ◽

Computational Efficiency ◽

Neural Nets ◽

Road Extraction ◽

Remote Sensing Images ◽

Feature Maps ◽

Aerial Vehicle

Roads are vital components of infrastructure, the extraction of which has become a topic of significant interest in the field of remote sensing. Because deep learning has been a popular method in image processing and information extraction, researchers have paid more attention to extracting road using neural networks. This article proposes the improvement of neural networks to extract roads from Unmanned Aerial Vehicle (UAV) remote sensing images. D-Linknet was first considered for its high performance; however, the huge scale of the net reduced computational efficiency. With a focus on the low computational efficiency problem of the popular D-LinkNet, this article made some improvements: (1) Replace the initial block with a stem block. (2) Rebuild the entire network based on ResNet units with a new structure, allowing for the construction of an improved neural network D-Linknetplus. (3) Add a 1 × 1 convolution layer before DBlock to reduce the input feature maps, reducing parameters and improving computational efficiency. Add another 1 × 1 convolution layer after DBlock to recover the required number of output channels. Accordingly, another improved neural network B-D-LinknetPlus was built. Comparisons were performed between the neural nets, and the verification were made with the Massachusetts Roads Dataset. The results show improved neural networks are helpful in reducing the network size and developing the precision needed for road extraction.

Download Full-text

Robotic grasp detection using a novel two-stage approach

ASP Transactions on Internet of Things ◽

10.52810/tiot.2021.100031 ◽

2021 ◽

Vol 1 (1) ◽

pp. 19-29

Author(s):

Zhe Chu ◽

Mengkai Hu ◽

Xiangyu Chen

Keyword(s):

Neural Network ◽

Neural Networks ◽

Network Models ◽

Particle Swarm Optimizer ◽

Neural Network Models ◽

Two Stage ◽

The Neural Network ◽

End To End ◽

Small Change ◽

Robotic Grasp

Recently, deep learning has been successfully applied to robotic grasp detection. Based on convolutional neural networks (CNNs), there have been lots of end-to-end detection approaches. But end-to-end approaches have strict requirements for the dataset used for training the neural network models and it’s hard to achieve in practical use. Therefore, we proposed a two-stage approach using particle swarm optimizer (PSO) candidate estimator and CNN to detect the most likely grasp. Our approach achieved an accuracy of 92.8% on the Cornell Grasp Dataset, which leaped into the front ranks of the existing approaches and is able to run at real-time speeds. After a small change of the approach, we can predict multiple grasps per object in the meantime so that an object can be grasped in a variety of ways.

Download Full-text

A Nonlinear Multiparameters Temperature Error Modeling and Compensation of POS Applied in Airborne Remote Sensing System

Journal of Applied Mathematics ◽

10.1155/2014/901539 ◽

2014 ◽

Vol 2014 ◽

pp. 1-11 ◽

Cited By ~ 2

Author(s):

Jianli Li ◽

Wenjian Wang ◽

Feng Jiao ◽

Jiancheng Fang ◽

Tao Yu

Keyword(s):

Neural Network ◽

Remote Sensing ◽

Error Modeling ◽

Compensation Method ◽

Temperature Error ◽

Bayesian Regularization ◽

Airborne Remote Sensing ◽

Accuracy Requirement ◽

Sensing System ◽

Single Temperature

The position and orientation system (POS) is a key equipment for airborne remote sensing systems, which provides high-precision position, velocity, and attitude information for various imaging payloads. Temperature error is the main source that affects the precision of POS. Traditional temperature error model is single temperature parameter linear function, which is not sufficient for the higher accuracy requirement of POS. The traditional compensation method based on neural network faces great problem in the repeatability error under different temperature conditions. In order to improve the precision and generalization ability of the temperature error compensation for POS, a nonlinear multiparameters temperature error modeling and compensation method based on Bayesian regularization neural network was proposed. The temperature error of POS was analyzed and a nonlinear multiparameters model was established. Bayesian regularization method was used as the evaluation criterion, which further optimized the coefficients of the temperature error. The experimental results show that the proposed method can improve temperature environmental adaptability and precision. The developed POS had been successfully applied in airborne TSMFTIS remote sensing system for the first time, which improved the accuracy of the reconstructed spectrum by 47.99%.

Download Full-text

Methods for maintenance of neural networks in continual learning scenarios

10.36227/techrxiv.14565078.v1 ◽

2021 ◽

Author(s):

Bhasker Sri Harsha Suri ◽

Manish Srivastava ◽

Kalidas Yeturu

Keyword(s):

Neural Network ◽

Neural Networks ◽

Production Systems ◽

Image Data ◽

Training Data ◽

Learning Loss ◽

Unseen Data ◽

Scoring Schemes ◽

The Difference ◽

Continual Learning

Neural networks suffer from catastrophic forgetting problem when deployed in a continual learning scenario where new batches of data arrive over time; however they are of different distributions from the previous data used for training the neural network. For assessing the performance of a model in a continual learning scenario, two aspects are important (i) to compute the difference in data distribution between a new and old batch of data and (ii) to understand the retention and learning behavior of deployed neural networks. Current techniques indicate the novelty of a new data batch by comparing its statistical properties with that of the old batch in the input space. However, it is still an open area of research to consider the perspective of a deployed neural network’s ability to generalize on the unseen data samples. In this work, we report a dataset distance measuring technique that indicates the novelty of a new batch of data while considering the deployed neural network’s perspective. We propose the construction of perspective histograms which are a vector representation of the data batches based on the correctness and confidence in the prediction of the deployed model. We have successfully tested the hypothesis empirically on image data coming MNIST Digits, MNIST Fashion, CIFAR10, for its ability to detect data perturbations of type rotation, Gaussian blur, and translation. Upon new data, given a model and its training data, we have proposed and evaluated four new scoring schemes, retention score (R), learning score (L), Oscore and SP-score for studying how much the model can retain its performance on past data, how much it can learn new data, the combined expression for the magnitude of retention and learning and stability-plasticity characteristics respectively. The scoring schemes have been evaluated MNIST Digits and MNIST Fashion data sets on different types of neural network architectures based on the number of parameters, activation functions, and learning loss functions, and an instance of a typical analysis report is presented. Machine learning model maintenance is a reality in production systems in the industry, and we hope our proposed methodology offers a solution to the need of the day in this aspect.

Download Full-text

CLASSIFICATION OF TREE SPECIES AND STANDING DEAD TREES BY FUSING UAV-BASED LIDAR DATA AND MULTISPECTRAL IMAGERY IN THE 3D DEEP NEURAL NETWORK POINTNET++

ISPRS Annals of Photogrammetry Remote Sensing and Spatial Information Sciences ◽

10.5194/isprs-annals-v-2-2020-203-2020 ◽

2020 ◽

Vol V-2-2020 ◽

pp. 203-210

Author(s):

S. Briechle ◽

P. Krzystek ◽

G. Vosselman

Keyword(s):

Neural Network ◽

Tree Species ◽

Deep Neural Network ◽

Lidar Data ◽

Individual Tree ◽

Dead Trees ◽

Standing Dead ◽

Standing Dead Trees ◽

Multiple Tree

Abstract. Knowledge of tree species mapping and of dead wood in particular is fundamental to managing our forests. Although individual tree-based approaches using lidar can successfully distinguish between deciduous and coniferous trees, the classification of multiple tree species is still limited in accuracy. Moreover, the combined mapping of standing dead trees after pest infestation is becoming increasingly important. New deep learning methods outperform baseline machine learning approaches and promise a significant accuracy gain for tree mapping. In this study, we performed a classification of multiple tree species (pine, birch, alder) and standing dead trees with crowns using the 3D deep neural network (DNN) PointNet++ along with UAV-based lidar data and multispectral (MS) imagery. Aside from 3D geometry, we also integrated laser echo pulse width values and MS features into the classification process. In a preprocessing step, we generated the 3D segments of single trees using a 3D detection method. Our approach achieved an overall accuracy (OA) of 90.2% and was clearly superior to a baseline method using a random forest classifier and handcrafted features (OA = 85.3%). All in all, we demonstrate that the performance of the 3D DNN is highly promising for the classification of multiple tree species and standing dead trees in practice.

Download Full-text

Investigating Semantic Augmentation in Virtual Environments for Image Segmentation Using Convolutional Neural Networks

Journal of Imaging ◽

10.3390/jimaging7080146 ◽

2021 ◽

Vol 7 (8) ◽

pp. 146

Author(s):

Joshua Ganter ◽

Simon Löffler ◽

Ron Metzger ◽

Katharina Ußling ◽

Christoph Müller

Keyword(s):

Neural Network ◽

Neural Networks ◽

Virtual Environments ◽

Semantic Information ◽

Synthetic Data ◽

Image Data ◽

Network Models ◽

Neural Network Models ◽

Real World Data ◽

Virtual Domain

Collecting real-world data for the training of neural networks is enormously time- consuming and expensive. As such, the concept of virtualizing the domain and creating synthetic data has been analyzed in many instances. This virtualization offers many possibilities of changing the domain, and with that, enabling the relatively fast creation of data. It also offers the chance to enhance necessary augmentations with additional semantic information when compared with conventional augmentation methods. This raises the question of whether such semantic changes, which can be seen as augmentations of the virtual domain, contribute to better results for neural networks, when trained with data augmented this way. In this paper, a virtual dataset is presented, including semantic augmentations and automatically generated annotations, as well as a comparison between semantic and conventional augmentation for image data. It is determined that the results differ only marginally for neural network models trained with the two augmentation approaches.

Download Full-text

AIRBORNE TECHNOLOGIES FOR DISASTER MANAGEMENT

ISPRS - International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences ◽

10.5194/isprs-archives-xlii-3-w8-387-2019 ◽

2019 ◽

Vol XLII-3/W8 ◽

pp. 387-393

Author(s):

J. Schulz

Keyword(s):

Remote Sensing ◽

Water Masses ◽

Aerial Images ◽

Airborne Remote Sensing ◽

Optimal Method ◽

Individual Tree ◽

Flood Peak ◽

Sensing Technology ◽

Aircraft System ◽

The Individual

Abstract. Currently, satellite-based systems and UAVs are very popular in the investigation of natural disasters. Both systems have their justification and advantages &ndash; but one should not forget the airborne remote sensing technology. The presentation shows with three examples very clearly how airborne remote sensing is still making great progress and in many cases represents the optimal method of data acquisition. The airborne detection of forest damages (especially currently the bark beetle in spruce stands) can determine the pest attack using CIR aerial images in combination with ALS and hyperspectral systems &ndash; down to the individual tree. Large forest areas of 100 sqkm and more can be recorded from planes on one day (100 sqkm with 10cm GSD on one day). Flood events &ndash; such as on the Elbe in 2013 &ndash; were recorded by many satellites. However, many evaluations require highresolution data (GSD 10cm), e.g. to clarify insurance claims. Here the aircraft system, which was able to fly below the cloud cover and was constantly flying at the height level of the flood peak, proved to be unbeatable. The phenomenon of urban flash floods is one of the consequences of climate change. Cities are not in a position to cope with the water masses of extreme rain events and so are confronted with major damages. In Germany, a number of cities are already preparing to manage short-term but extreme water masses. The complicated hydrographic and hydraulic calculations and simulations require above all one thing &ndash; a precise data basis. This involves, for example, the height of kerbstones and the recording of every gully and every obstacle. Such city-wide data can only be collected effectively by photogrammetric analysis of aerial photography (GSD 5 to 10cm).

Download Full-text