Exploiting Scalable Parallelism for Remote Sensing Analysis Models by Data Transformation Graph

RDCRMG: A Raster Dataset Clean & Reconstitution Multi-Grid Architecture for Remote Sensing Monitoring of Vegetation Dryness

Remote Sensing ◽

10.3390/rs10091376 ◽

2018 ◽

Vol 10 (9) ◽

pp. 1376 ◽

Cited By ~ 9

Author(s):

Sijing Ye ◽

Diyou Liu ◽

Xiaochuang Yao ◽

Huaizhi Tang ◽

Quan Xiong ◽

...

Keyword(s):

Remote Sensing ◽

Data Storage ◽

Large Scale ◽

High Efficiency ◽

Data Extraction ◽

Automatic Monitoring ◽

Information Distortion ◽

Remote Sensing Monitoring ◽

Data Inversion ◽

Analysis Models

In recent years, remote sensing (RS) research on crop growth status monitoring has gradually turned from static spectrum information retrieval in large-scale to meso-scale or micro-scale, timely multi-source data cooperative analysis; this change has presented higher requirements for RS data acquisition and analysis efficiency. How to implement rapid and stable massive RS data extraction and analysis becomes a serious problem. This paper reports on a Raster Dataset Clean & Reconstitution Multi-Grid (RDCRMG) architecture for remote sensing monitoring of vegetation dryness in which different types of raster datasets have been partitioned, organized and systematically applied. First, raster images have been subdivided into several independent blocks and distributed for storage in different data nodes by using the multi-grid as a consistent partition unit. Second, the “no metadata model” ideology has been referenced so that targets raster data can be speedily extracted by directly calculating the data storage path without retrieving metadata records; third, grids that cover the query range can be easily assessed. This assessment allows the query task to be easily split into several sub-tasks and executed in parallel by grouping these grids. Our RDCRMG-based change detection of the spectral reflectance information test and the data extraction efficiency comparative test shows that the RDCRMG is reliable for vegetation dryness monitoring with a slight reflectance information distortion and consistent percentage histograms. Furthermore, the RDCGMG-based data extraction in parallel circumstances has the advantages of high efficiency and excellent stability compared to that of the RDCGMG-based data extraction in serial circumstances and traditional data extraction. At last, an RDCRMG-based vegetation dryness monitoring platform (VDMP) has been constructed to apply RS data inversion in vegetation dryness monitoring. Through actual applications, the RDCRMG architecture is proven to be appropriate for timely vegetation dryness RS automatic monitoring with better performance, more reliability and higher extensibility. Our future works will focus on integrating more kinds of continuously updated RS data into the RDCRMG-based VDMP and integrating more multi-source datasets based collaborative analysis models for agricultural monitoring.

Download Full-text

Scientific workflow scheduling based on data transformation graph for remote sensing application

10.5194/egusphere-egu21-1255 ◽

2021 ◽

Author(s):

Zhuojing tian ◽

Zhenchun huang ◽

Yinong zhang ◽

Yanwei zhao ◽

En fu ◽

...

Keyword(s):

Remote Sensing ◽

Genetic Algorithm ◽

Data Transformation ◽

Scientific Workflow ◽

Workflow Scheduling ◽

Cloud Environment ◽

Sensing Applications ◽

Random Algorithm ◽

Remote Sensing Applications ◽

Sensing Application

Abstract: As the amount of data and computation of scientific workflow applications continue to grow, distributed and heterogeneous computing infrastructures such as inter-cloud environments provide this type of application with a great number of computing resources to meet corresponding needs. In the inter-cloud environment, how to effectively map tasks to cloud service providers to meet QoS(quality of service) constraints based on user requirements has become an important research direction. Remote sensing applications need to process terabytes of data each time, however frequent and huge data transmission across the cloud will bring huge performance bottlenecks for execution, and seriously affect the result of QoS constraints such as makespan and cost. Using a data transformation graph(DTG) to study the data transfer process of global drought detection application, the specific optimization strategy is obtained based on the characteristics of application and environment, and according to this, one inter-cloud workflow scheduling method based on genetic algorithm is proposed, under the condition of satisfying the user&#8217;s QoS constraints, the makespan the cost can be minimized. The experimental results show that compared with the standard genetic algorithm, random algorithm, random algorithm, and round-robin algorithm, the optimized genetic algorithm can greatly improve the scheduling performance of data computation-intensive scientific workflows such as remote sensing applications and reduce the impact of performance bottlenecks.Keywords: scientific workflow scheduling; inter-cloud environment; remote sensing application; data transformation graph;

Download Full-text