Fusion neural networks for plant classification: learning to combine RGB, hyperspectral, and lidar data

Airborne remote sensing offers unprecedented opportunities to efficiently monitor vegetation, but methods to delineate and classify individual plant species using the collected data are still actively being developed and improved. The Integrating Data science with Trees and Remote Sensing (IDTReeS) plant identification competition openly invited scientists to create and compare individual tree mapping methods. Participants were tasked with training taxon identification algorithms based on two sites, to then transfer their methods to a third unseen site, using field-based plant observations in combination with airborne remote sensing image data products from the National Ecological Observatory Network (NEON). These data were captured by a high resolution digital camera sensitive to red, green, blue (RGB) light, hyperspectral imaging spectrometer spanning the visible to shortwave infrared wavelengths, and lidar systems to capture the spectral and structural properties of vegetation. As participants in the IDTReeS competition, we developed a two-stage deep learning approach to integrate NEON remote sensing data from all three sensors and classify individual plant species and genera. The first stage was a convolutional neural network that generates taxon probabilities from RGB images, and the second stage was a fusion neural network that “learns” how to combine these probabilities with hyperspectral and lidar data. Our two-stage approach leverages the ability of neural networks to flexibly and automatically extract descriptive features from complex image data with high dimensionality. Our method achieved an overall classification accuracy of 0.51 based on the training set, and 0.32 based on the test set which contained data from an unseen site with unknown taxa classes. Although transferability of classification algorithms to unseen sites with unknown species and genus classes proved to be a challenging task, developing methods with openly available NEON data that will be collected in a standardized format for 30 years allows for continual improvements and major gains for members of the computational ecology community. We outline promising directions related to data preparation and processing techniques for further investigation, and provide our code to contribute to open reproducible science efforts.

SIOS’s airborne remote sensing campaigns in Svalbard

<p>Svalbard Integrated Arctic Earth Observing System (SIOS) is an international collaboration of 24 research institutions from 9 countries studying the environment and climate in and around Svalbard. The global pandemic of Coronavirus disease (Covid-19) has affected the Svalbard research in a number of ways due to nationwide lockdown in many countries, strict travel restrictions in Svalbard, and quarantine regulations. Many field campaigns to Svalbard were cancelled in 2020 and campaigns in 2021 are still uncertain. In response to this challenge, we conducted practical activities to support the Svalbard science community in filling gaps in scientific observations. One of our activities involved conducting airborne remote sensing campaigns in Svalbard to support scientific projects. In 2020, SIOS supported 10 scientific projects by conducting 25 hours of aircraft and unmanned aerial vehicle (UAV)-based data collection in Svalbard. This is one of the finest ways to fill the data gap in the current situation as it is practically possible to conduct field campaigns using airborne platforms in spite of travel restrictions. We are using the aerial camera and hyperspectral sensor installed onboard the Dornier DO228 aircraft operated by the local company Lufttransport to acquire aerial images and hyperspectral data from various locations in Svalbard. The hyperspectral sensor image the ground in 186 spectral bands covering the range 400-1000 nm. Hyperspectral data can be used to map and characterise earth, ice and ocean surface features, such as minerals, vegetation, glaciers and snow cover, colour and pollutants. Further, it can be used to make 3D models of the terrain as well as searching for the presence of animals (e.g. counting seals). In addition, aerial photos are particularly useful tool to follow the seasonal dynamic changes and extent in sea ice cover, tracking icebergs, ocean productivity (Chlorophyll a) and river runoff (turbidity). Data collected from the SIOS funded airborne missions will not only help to fill a few of the data gaps resulting from the lockdown but also will be used by glaciologists, biologists, hydrologists, and other Earth system scientists to understand the state of the environment of Svalbard during these times. In 2021, we are continuing this activity by conducting more airborne campaigns in Svalbard. In this presentation, we will specifically focus on the overview of projects supported by airborne remote sensing campaigns.</p>