scholarly journals On-the-Fly Fusion of Remotely-Sensed Big Data Using an Elastic Computing Paradigm with a Containerized Spark Engine on Kubernetes

Sensors ◽  
2021 ◽  
Vol 21 (9) ◽  
pp. 2971
Author(s):  
Wei Huang ◽  
Jianzhong Zhou ◽  
Dongying Zhang
Keyword(s):  
Cloud Computing ◽  
Big Data ◽  
Satellite Image ◽  
Remotely Sensed ◽  
Observation Data ◽  
Fusion Model ◽  
Computing Paradigm ◽  
Big Picture ◽  
Elastic Computing ◽  
Earth Observation Data

Remotely-sensed satellite image fusion is indispensable for the generation of long-term gap-free Earth observation data. While cloud computing (CC) provides the big picture for RS big data (RSBD), the fundamental question of the efficient fusion of RSBD on CC platforms has not yet been settled. To this end, we propose a lightweight cloud-native framework for the elastic processing of RSBD in this study. With the scaling mechanisms provided by both the Infrastructure as a Service (IaaS) and Platform as a Services (PaaS) of CC, the Spark-on-Kubernetes operator model running in the framework can enhance the efficiency of Spark-based algorithms without considering bottlenecks such as task latency caused by an unbalanced workload, and can ease the burden to tune the performance parameters for their parallel algorithms. Internally, we propose a task scheduling mechanism (TSM) to dynamically change the Spark executor pods’ affinities to the computing hosts. The TSM learns the workload of a computing host. Learning from the ratio between the number of completed and failed tasks on a computing host, the TSM dispatches Spark executor pods to newer and less-overwhelmed computing hosts. In order to illustrate the advantage, we implement a parallel enhanced spatial and temporal adaptive reflectance fusion model (PESTARFM) to enable the efficient fusion of big RS images with a Spark aggregation function. We construct an OpenStack cloud computing environment to test the usability of the framework. According to the experiments, TSM can improve the performance of the PESTARFM using only PaaS scaling to about 11.7%. When using both the IaaS and PaaS scaling, the maximum performance gain with the TSM can be even greater than 13.6%. The fusion of such big Sentinel and PlanetScope images requires less than 4 min in the experimental environment.

scholarly journals Enabling the Big Earth Observation Data via Cloud Computing and DGGS: Opportunities and Challenges

2019 ◽  
Vol 12 (1) ◽  
pp. 62 ◽  
Author(s):  
Xiaochuang Yao ◽  
Guoqing Li ◽  
Junshi Xia ◽  
Jin Ben ◽  
Qianqian Cao ◽  
...  
Keyword(s):  
Cloud Computing ◽  
Big Data ◽  
Earth Observation ◽  
Current Status ◽  
Observation Data ◽  
Computing Technology ◽  
Grid Systems ◽  
Data Framework ◽  
Earth Observation Data ◽  
Global Grid

In the era of big data, the explosive growth of Earth observation data and the rapid advancement in cloud computing technology make the global-oriented spatiotemporal data simulation possible. These dual developments also provide advantageous conditions for discrete global grid systems (DGGS). DGGS are designed to portray real-world phenomena by providing a spatiotemporal unified framework on a standard discrete geospatial data structure and theoretical support to address the challenges from big data storage, processing, and analysis to visualization and data sharing. In this paper, the trinity of big Earth observation data (BEOD), cloud computing, and DGGS is proposed, and based on this trinity theory, we explore the opportunities and challenges to handle BEOD from two aspects, namely, information technology and unified data framework. Our focus is on how cloud computing and DGGS can provide an excellent solution to enable big Earth observation data. Firstly, we describe the current status and data characteristics of Earth observation data, which indicate the arrival of the era of big data in the Earth observation domain. Subsequently, we review the cloud computing technology and DGGS framework, especially the works and contributions made in the field of BEOD, including spatial cloud computing, mainstream big data platform, DGGS standards, data models, and applications. From the aforementioned views of the general introduction, the research opportunities and challenges are enumerated and discussed, including EO data management, data fusion, and grid encoding, which are concerned with analysis models and processing performance of big Earth observation data with discrete global grid systems in the cloud environment.

An Efficient and Scalable Framework for Processing Remotely Sensed Big Data in Cloud Computing Environments

2019 ◽  
Vol 57 (7) ◽  
pp. 4294-4308 ◽  
Author(s):  
Jin Sun ◽  
Yi Zhang ◽  
Zebin Wu ◽  
Yaoqin Zhu ◽  
Xianliang Yin ◽  
...  
Keyword(s):  
Cloud Computing ◽  
Big Data ◽  
Remotely Sensed ◽  
Computing Environments

scholarly journals FluxEngine: A Flexible Processing System for Calculating Atmosphere–Ocean Carbon Dioxide Gas Fluxes and Climatologies

2016 ◽  
Vol 33 (4) ◽  
pp. 741-756 ◽  
Author(s):  
Jamie D. Shutler ◽  
Peter E. Land ◽  
Jean-Francois Piolle ◽  
David K. Woolf ◽  
Lonneke Goddijn-Murphy ◽  
...  
Keyword(s):  
Carbon Dioxide ◽  
Cloud Computing ◽  
Processing System ◽  
Earth Observation ◽  
Observation Data ◽  
Large Spatial Scale ◽  
Process Indicator ◽  
Flux Calculation ◽  
Gas Fluxes ◽  
Earth Observation Data

AbstractThe air–sea flux of greenhouse gases [e.g., carbon dioxide (CO2)] is a critical part of the climate system and a major factor in the biogeochemical development of the oceans. More accurate and higher-resolution calculations of these gas fluxes are required if researchers are to fully understand and predict future climate. Satellite Earth observation is able to provide large spatial-scale datasets that can be used to study gas fluxes. However, the large storage requirements needed to host such data can restrict its use by the scientific community. Fortunately, the development of cloud computing can provide a solution. This paper describes an open-source air–sea CO2 flux processing toolbox called the “FluxEngine,” designed for use on a cloud-computing infrastructure. The toolbox allows users to easily generate global and regional air–sea CO2 flux data from model, in situ, and Earth observation data, and its air–sea gas flux calculation is user configurable. Its current installation on the Nephalae Cloud allows users to easily exploit more than 8 TB of climate-quality Earth observation data for the derivation of gas fluxes. The resultant netCDF data output files contain >20 data layers containing the various stages of the flux calculation along with process indicator layers to aid interpretation of the data. This paper describes the toolbox design, which verifies the air–sea CO2 flux calculations; demonstrates the use of the tools for studying global and shelf sea air–sea fluxes; and describes future developments.

scholarly journals Integration of Earth Observation Data and Spatial Approach to Delineate and Manage Aeolian Sand-Affected Wasteland in Highly Productive Lands of Haryana, India

2018 ◽  
Vol 2018 ◽  
pp. 1-7 ◽  
Author(s):  
Kishan Singh Rawat ◽  
Shashi Vind Mishra ◽  
Sudhir Kumar Singh
Keyword(s):  
Accuracy Assessment ◽  
Satellite Image ◽  
Geographical Area ◽  
Integrated Approach ◽  
Observation Data ◽  
Visual Interpretation ◽  
Aeolian Sand ◽  
Spatial Approach ◽  
Total Geographical Area ◽  
Earth Observation Data

The western part of the country India is surrounded by Thar desert. Due to climate change, many regions in the world are facing different challenges. The objective of the study was to quantify the aeolian sand-affected land through integrated approach. The LANDSAT-ETM+ satellite image of 2009 has been used to distinguish recently affected areas by aeolian sand. A combined approach of digital classification backed with visual interpretation and ground verification was adopted. In addition to classification accuracy assessment was performed using field observations. Evidence based results of aeolian sand-affected areas have suggested that wasteland area has increased up to 4,427.55 ha (6.79%) of total geographical area. Two types of aeolian sands areas have been detected, namely, moderately affected (3,881.77 ha) and severely affected (545.79 ha). Moderately and severely affected aeolian soil lands have been more accurately mapped with reasonably good accuracy whereas smaller aeolian affected areas within croplands are mapped with low accuracy. The present study provides easy methodology for delineation, classification, and characterization of aeolian affected sands.

scholarly journals Remote Sensing

2021 ◽  
pp. 49-61
Author(s):  
Miguel Ángel Esbrí
Keyword(s):  
Remote Sensing ◽  
Big Data ◽  
Information Access ◽  
Open Data ◽  
Earth Observation ◽  
Observation Data ◽  
Related Data ◽  
Data Formats ◽  
Earth Observation Data ◽  
Access Services

AbstractIn this chapter we present the concepts of remote sensing and Earth Observation and, explain why several of their characteristics (volume, variety and velocity) make us consider Earth Observation as Big Data. Thereafter, we discuss the most commonly open data formats used to store and share the data. The main sources of Earth Observation data are also described, with particular focus on the constellation of Sentinel satellites, Copernicus Hub and its six thematic services, as well as other private initiatives like the five Copernicus-related Data and Information Access Services and  Sentinel Hub. Next, we present an overview of representative software technologies for efficiently describing, storing, querying and accessing Earth Observation datasets. The chapter concludes with a summary of the Earth Observation datasets used in each DataBio pilot.

SAMEVED

2013 ◽  
Vol 5 (2) ◽  
pp. 27-42 ◽  
Author(s):  
Shao-Jui Chen ◽  
Jing-Ying Huang ◽  
Cheng-Ta Huang ◽  
Wei-Jen Wang
Keyword(s):  
Cloud Computing ◽  
Virtual Machines ◽  
Cloud Service ◽  
Web Interface ◽  
Computing Paradigm ◽  
Elastic Computing ◽  
Virtual Routers ◽  
User Friendly ◽  
Cloud Applications ◽  
Remote Invocation

Cloud computing is an emerging computing paradigm that provides all kinds of services through the Internet. Existing elastic computing approaches are popular in cloud computing. They can fulfill the requirements of some cloud applications, but usually fail to provide an isolated computing environment consisting of connected virtual machines over a user-defined network topology. This paper presents a system architecture, namely SAMEVED, which exposes a cloud service that can allocate and manage a private, virtual elastic datacenter by integrating VPN and virtual routers into existing virtualization technologies. Authentication is required by user login while using SAMEVED. A user-friendly web interface and remote invocation interface are provided to support different operations for different users with different privileges.

scholarly journals GEE TIMESERIES EXPLORER FOR QGIS – INSTANT ACCESS TO PETABYTES OF EARTH OBSERVATION DATA

2021 ◽  
Vol XLVI-4/W2-2021 ◽  
pp. 155-158
Author(s):  
P. Rufin ◽  
A. Rabe ◽  
L. Nill ◽  
P. Hostert
Keyword(s):  
Time Series ◽  
Time Series Data ◽  
Satellite Image ◽  
Google Earth ◽  
Earth Observation ◽  
Series Data ◽  
Observation Data ◽  
Local Data ◽  
Small Areas ◽  
Earth Observation Data

Abstract. Earth observation analysis workflows commonly require mass processing of time series data, with data volumes easily exceeding terabyte magnitude, even for relatively small areas of interest. Cloud processing platforms such as Google Earth Engine (GEE) leverage accessibility to satellite image archives and thus facilitate time series analysis workflows. Instant visualization of time series data and integration with local data sources is, however, currently not implemented or requires coding customized scripts or applications. Here, we present the GEE Timeseries Explorer plugin which grants instant access to GEE from within QGIS. It seamlessly integrates the QGIS user interface with a compact widget for visualizing time series from any predefined or customized GEE image collection. Users can visualize time series profiles for a given coordinate as an interactive plot or visualize images with customized band rendering within the QGIS map canvas. The plugin is available through the QGIS plugin repository and detailed documentation is available online (https://geetimeseriesexplorer.readthedocs.io/).

Sign in / Sign up

Export Citation Format

Share Document