scholarly journals DINCAE 1.0: a convolutional neural network with error estimates to reconstruct sea surface temperature satellite observations

2019 ◽  
Author(s):  
Alexander Barth ◽  
Aida Alvera-Azcárate ◽  
Matjaz Licer ◽  
Jean-Marie Beckers
Keyword(s):  
Neural Network ◽  
Missing Data ◽  
Surface Temperature ◽  
Sea Surface Temperature ◽  
Satellite Data ◽  
World Ocean ◽  
Error Variance ◽  
Empirical Orthogonal Functions ◽  
Satellite Observations ◽  
Sea Surface

Abstract. A method to reconstruct missing data in satellite data using a neural network is presented. Satellite observations working in the optical and infrared bands are affected by clouds, which obscure part of the ocean underneath. In this paper, a neural network with the structure of a convolutional auto-encoder is developed to reconstruct the missing data based on the available cloud-free pixels in satellite images. However, it is unclear how to handle missing data (or data with variable accuracy) in a neural network when using incomplete satellite data in the training phase. The present work shows a consistent approach which uses essentially the satellite data and its expected error variance as input and provides the reconstructed field along with its expected error variance as output. The neural network is trained by maximizing likelihood of the observed value. The approach, called DINCAE (Data-Interpolating Convolutional Auto-Encoder) is applied to a relatively long time-series of Advanced Very High Resolution Radiometer (AVHRR) sea surface temperature data and compared to DINEOF (Data Interpolating Empirical Orthogonal Functions), a method to reconstruct missing data based on an EOF decomposition. The reconstruction error of both approaches is computed using cross-validation and in situ observations from the World Ocean Database. DINCAE results have lower error, while showing higher variability than the DINEOF reconstruction.

A convolutional neural network with error estimates to reconstruct sea surface temperature satellite observations (DINCAE)

2020 ◽  
Author(s):  
Alexander Barth ◽  
Aida Alvera Azcárate ◽  
Matjaz Licer ◽  
Jean-Marie Beckers
Keyword(s):  
Neural Network ◽  
Missing Data ◽  
Surface Temperature ◽  
Sea Surface Temperature ◽  
Error Estimates ◽  
Satellite Data ◽  
Cross Validation ◽  
Error Variance ◽  
Satellite Observations ◽  
Sea Surface

<p>A method to reconstruct missing data in satellite data using a neural network is presented. Satellite observations working in the optical and infrared bands are affected by clouds, which obscure part of the ocean underneath. In this paper, a neural network with the structure of a convolutional auto-encoder is developed to reconstruct the missing data based on the available cloud-free pixels in satellite images. However, it is unclear how to handle missing data (or data with variable accuracy) in a neural network when using incomplete satellite data in the training phase. The present work shows a consistent approach which uses essentially the satellite data and its expected error variance as input and provides the reconstructed field along with its expected error variance as output. The approach is motivated by the way models and observations are combined in the frame of data assimilation. The neural network is trained by maximizing the likelihood of the observed value. The corresponding error variances are estimated during training and do not need to be known a priori. The approach, called DINCAE (Data-Interpolating Convolutional Auto-Encoder) is applied to a relatively long time-series of Advanced Very High Resolution Radiometer (AVHRR) sea surface temperature data and compared to DINEOF (Data Interpolating Empirical Orthogonal Functions), a method to reconstruct missing data based on an EOF decomposition. The reconstruction error of both approaches is computed using cross-validation and in situ observations from the World Ocean Database. DINCAE results have lower error, while showing higher variability than the DINEOF reconstruction. The resulting error estimates are also validated using the cross-validation data and they follow closely the expected Gaussian distribution.</p>

scholarly journals DINCAE 1.0: a convolutional neural network with error estimates to reconstruct sea surface temperature satellite observations

2020 ◽  
Vol 13 (3) ◽  
pp. 1609-1622 ◽  
Author(s):  
Alexander Barth ◽  
Aida Alvera-Azcárate ◽  
Matjaz Licer ◽  
Jean-Marie Beckers
Keyword(s):  
Neural Network ◽  
Missing Data ◽  
Surface Temperature ◽  
Sea Surface Temperature ◽  
Error Variance ◽  
Temperature Data ◽  
Satellite Observations ◽  
Sea Surface ◽  
Function Decomposition ◽  
Surface Temperature Data

Abstract. A method to reconstruct missing data in sea surface temperature data using a neural network is presented. Satellite observations working in the optical and infrared bands are affected by clouds, which obscure part of the ocean underneath. In this paper, a neural network with the structure of a convolutional auto-encoder is developed to reconstruct the missing data based on the available cloud-free pixels in satellite images. Contrary to standard image reconstruction with neural networks, this application requires a method to handle missing data (or data with variable accuracy) in the training phase. The present work shows a consistent approach which uses the satellite data and its expected error variance as input and provides the reconstructed field along with its expected error variance as output. The neural network is trained by maximizing the likelihood of the observed value. The approach, called DINCAE (Data INterpolating Convolutional Auto-Encoder), is applied to a 25-year time series of Advanced Very High Resolution Radiometer (AVHRR) sea surface temperature data and compared to DINEOF (Data INterpolating Empirical Orthogonal Functions), a commonly used method to reconstruct missing data based on an EOF (empirical orthogonal function) decomposition. The reconstruction error of both approaches is computed using cross-validation and in situ observations from the World Ocean Database. DINCAE results have lower error while showing higher variability than the DINEOF reconstruction.

scholarly journals Impacts of Sea Surface Temperature Uncertainty on the Western North Pacific Subtropical High (WNPSH) and Rainfall

2011 ◽  
Vol 26 (3) ◽  
pp. 371-387 ◽  
Author(s):  
Xiaodong Hong ◽  
Craig H. Bishop ◽  
Teddy Holt ◽  
Larry O’Neill
Keyword(s):  
Surface Temperature ◽  
Sea Surface Temperature ◽  
North Pacific ◽  
Western North Pacific ◽  
Forecast Error ◽  
Error Variance ◽  
Satellite Observations ◽  
Sea Surface ◽  
Subtropical High ◽  
Forecast Error Variance

Abstract This paper examines the sensitivity of short-term forecasts of the western North Pacific subtropical high (WNPSH) and rainfall to sea surface temperature (SST) uncertainty using the Coupled Ocean–Atmosphere Mesoscale Prediction System (COAMPS). A comparison of analyzed SSTs with satellite observations of SST indicates that SST analysis errors are particularly pronounced on horizontal scales from 100 to 200 km, similar to the mesoscale eddy scales in the Kuroshio region. Since significant oceanic variations occur on these scales, it is of interest to examine the effects of representing this small-scale uncertainty with random, scale-dependent perturbations. An SST ensemble perturbation generation technique is used here that enables temporal and spatial correlations to be controlled and produces initial SST fields comparable to satellite observations. The atmospheric model develops large uncertainty in the Korea and Japan area due to the fluctuation in the horizontal pressure gradient caused by the location of the WNPSH. This, in turn, increases the variance of the low-level jet (LLJ) over southeast China, resulting in large differences in the moist transport flux from the tropical ocean and subsequent rainfall. Validation using bin-mean statistics shows that the ensemble forecast with the perturbed SST better distinguishes large forecast error variance from small forecast error variance. The results suggest that using the SST perturbation as a proxy for the ocean ensemble in a coupled atmosphere and ocean ensemble system is feasible and computationally efficient.

Filling in missing sea-surface temperature satellite data over the Eastern Mediterranean Sea using the DINEOF algorithm

2014 ◽  
Vol 6 (1) ◽  
Author(s):  
Andreas Nikolaidis ◽  
Georgios Georgiou ◽  
Diofantos Hadjimitsis ◽  
Evangelos Akylas
Keyword(s):  
Missing Data ◽  
Surface Temperature ◽  
Sea Surface Temperature ◽  
Eastern Mediterranean ◽  
Western Mediterranean ◽  
Empirical Orthogonal Functions ◽  
Sea Surface ◽  
Temperature Fields ◽  
Special Technique ◽  
Orthogonal Functions

AbstractThe Data Interpolating Empirical Orthogonal Functions method is a special technique based on Empirical Orthogonal Functions and developed to reconstruct missing data from satellite images, which is especially useful for filling in missing data from geophysical fields. Successful experiments in the Western Mediterranean encouraged extension of the application eastwards using a similar experimental implementation. The present study summarizes the experimental work done, the implementation of the method and its ability to reconstruct the sea-surface temperature fields over the Eastern Mediterranean basin, and specifically in the Levantine Sea. L3 type Satellite Sea-surface Temperature data has been used and reprocessed in order to recover missing information from cloudy images. Data reconstruction with this method proved to be extremely effective, even when using a relatively small number of time steps, and markedly accelerated the procedure. A detailed comparison with the two oceanographic models proves the accuracy of the method and the validity of the reconstructed fields.

scholarly journals DINEOF reconstruction for creating long-term cloud-free sea surface temperature data records: A Case study in Lombok Strait, Indonesia

2021 ◽  
Vol 893 (1) ◽  
pp. 012069
Author(s):  
Yochi Okta Andrawina ◽  
Ratu Almira Kismawardhani ◽  
Hasti Amrih Rejeki
Keyword(s):  
Missing Data ◽  
Climate Variability ◽  
Surface Temperature ◽  
Sea Surface Temperature ◽  
Satellite Image ◽  
Empirical Orthogonal Functions ◽  
Sea Surface ◽  
Series Data ◽  
Data Record

Abstract A long-term reliable sea surface temperature (SST) satellite data record is requisite resources for monitoring to understand climate variability. Creating a long-term data record especially for climate variability requires a combination of multiple satellite products. Consequently, missing data issues are inevitable. Hence, DINEOF (Data Interpolating Empirical Orthogonal Functions) has been applied to attain a complete and coherent multi-sensor SST data record with EOF-based technique by reconstructing the missing data. Unfortunately, the technique can lead to large discontinuities in the data reconstruction due to images depiction within long time series data. For that reason, filtering the temporal covariance matrix had been applied to reduce the spurious variability and more realistic reconstructions are obtained. However, this approach has not yet tested in tropical region with higher evaporation which cause incomplete satellite image coverage. Therefore, the objective of this research is to reconstruct SST of Lombok strait with data gaps up to 58.16% in one year. It is successfully reconstructed until the last iteration of 42 optimal EOF modes with the convergence achieved up to 0.9806×10-3, including previous set-aside data for internal cross-validation. The results highlight that the DINEOF method can effectively reconstruct SST data in Lombok Strait.

scholarly journals Enhancing temporal correlations in EOF expansions for the reconstruction of missing data using DINEOF

2009 ◽  
Vol 6 (2) ◽  
pp. 1547-1568 ◽  
Author(s):  
A. Alvera-Azcárate ◽  
A. Barth ◽  
D. Sirjacobs ◽  
J.-M. Beckers
Keyword(s):  
Missing Data ◽  
Surface Temperature ◽  
Sea Surface Temperature ◽  
Empirical Orthogonal Functions ◽  
Sea Surface ◽  
The Black Sea ◽  
Time Variability ◽  
Orthogonal Functions ◽  
Data Set ◽  
Temporal Correlations

Abstract. DINEOF (Data Interpolating Empirical Orthogonal Functions) is an EOF-based technique for the reconstruction of missing data in geophysical fields, such as those produced by clouds in sea surface temperature satellite images. A technique to reduce spurious time variability in DINEOF reconstructions is presented. The reconstruction of these images within a long time series using DINEOF can lead to large discontinuities in the reconstruction. Filtering the temporal covariance matrix allows to reduce this spurious variability and therefore more realistic reconstructions are obtained. The approach is tested in a three years sea surface temperature data set over the Black Sea. The effect of the filter in the temporal EOFs is presented, as well as some examples of the improvement achieved with the filtering in the SST reconstruction, both compared to the DINEOF approach without filtering.

scholarly journals Enhancing temporal correlations in EOF expansions for the reconstruction of missing data using DINEOF

Ocean Science ◽  
2009 ◽  
Vol 5 (4) ◽  
pp. 475-485 ◽  
Author(s):  
A. Alvera-Azcárate ◽  
A. Barth ◽  
D. Sirjacobs ◽  
J.-M. Beckers
Keyword(s):  
Missing Data ◽  
Surface Temperature ◽  
Sea Surface Temperature ◽  
Empirical Orthogonal Functions ◽  
Sea Surface ◽  
The Black Sea ◽  
Time Variability ◽  
Orthogonal Functions ◽  
Data Set ◽  
Temporal Correlations

Abstract. DINEOF (Data Interpolating Empirical Orthogonal Functions) is an EOF-based technique for the reconstruction of missing data in geophysical fields, such as those produced by clouds in sea surface temperature satellite images. A technique to reduce spurious time variability in DINEOF reconstructions is presented. The reconstruction of these images within a long time series using DINEOF can lead to large discontinuities in the reconstruction. Filtering the temporal covariance matrix allows to reduce this spurious variability and therefore more realistic reconstructions are obtained. The approach is tested in a three years sea surface temperature data set over the Black Sea. The effect of the filter in the temporal EOFs is presented, as well as some examples of the improvement achieved with the filtering in the SST reconstruction, both compared to the DINEOF approach without filtering.

scholarly journals DINCAE 2: multivariate convolutional neural network with error estimates to reconstruct sea surface temperature satellite and altimetry observations

2021 ◽  
Author(s):  
Alexander Barth ◽  
Aida Alvera-Azcárate ◽  
Charles Troupin ◽  
Jean-Marie Beckers
Keyword(s):  
Neural Network ◽  
Missing Data ◽  
Surface Temperature ◽  
Sea Surface Temperature ◽  
Cost Function ◽  
Chlorophyll Concentration ◽  
Sea Surface ◽  
Bayesian Optimization ◽  
Superior Performance ◽  
The Neural Network

Abstract. DINCAE (Data INterpolating Convolutional Auto-Encoder) is a neural network to reconstruct missing data (e.g. obscured by clouds or gaps between tracks) in satellite data. Contrary to standard image reconstruction (in-painting) with neural networks, this application requires a method to handle missing data (or data with variable accuracy) already in the training phase. Instead of using a standard L2 (or L1) cost function, the neural network (U-Net type of network) is optimized by minimizing the negative log likelihood assuming a Gaussian distribution (characterized by a mean and a variance). As a consequence, the neural network also provides an expected error variance of the reconstructed field (per pixel and per time instance). In this updated version DINCAE 2.0, the code was rewritten in Julia and a new type of skip connection has been implemented which showed superior performance with respect to the previous version. The method has also been extended to handle multivariate data (an example will be shown with sea-surface temperature, chlorophyll concentration and wind fields). The improvement of this network is demonstrated in the Adriatic Sea. Convolutional networks work usually with gridded data as input. This is however a limitation for some data types used in oceanography and in Earth Sciences in general, where observations are often irregularly sampled. The first layer of the neural network and the cost function have been modified so that unstructured data can also be used as inputs to obtain gridded fields as output. To demonstrate this, the neural network is applied to along-track altimetry data in the Mediterranean Sea. Results from a 20-year reconstruction are presented and validated. Hyperparameters are determined using Bayesian optimization and minimizing the error relative to a development dataset.

Sign in / Sign up

Export Citation Format

Share Document