RSIMS: Large-Scale Heterogeneous Remote Sensing Images Management System

With the remarkable development and progress of earth-observation techniques, remote sensing data keep growing rapidly and their volume has reached exabyte scale. However, it's still a big challenge to manage and process such huge amounts of remote sensing data with complex and diverse structures. This paper designs and realizes a distributed storage system for large-scale remote sensing data storage, access, and retrieval, called RSIMS (remote sensing images management system), which is composed of three sub-modules: RSIAPI, RSIMeta, RSIData. Structured text metadata of different remote sensing images are all stored in RSIMeta based on a set of uniform models, and then indexed by the distributed multi-level Hilbert grids for high spatiotemporal retrieval performance. Unstructured binary image files are stored in RSIData, which provides large scalable storage capacity and efficient GDAL (Geospatial Data Abstraction Library) compatible I/O interfaces. Popular GIS software and tools (e.g., QGIS, ArcGIS, rasterio) can access data stored in RSIData directly. RSIAPI provides users a set of uniform interfaces for data access and retrieval, hiding the complex inner structures of RSIMS. The test results show that RSIMS can store and manage large amounts of remote sensing images from various sources with high and stable performance, and is easy to deploy and use.

Download Full-text

RESEARCH ON MULTI-SOURCE SATELLITE IMAGE DATABASE MANAGEMENT SYSTEM

ISPRS - International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences ◽

10.5194/isprs-archives-xlii-3-w10-565-2020 ◽

2020 ◽

Vol XLII-3/W10 ◽

pp. 565-568

Author(s):

N. Fu ◽

L. Sun ◽

H. Z. Yang ◽

J. Ma ◽

B. Q. Liao

Keyword(s):

Remote Sensing ◽

Data Storage ◽

Management System ◽

Satellite Image ◽

Remote Sensing Data ◽

Database Management System ◽

Sensor Data ◽

Remote Sensing Images ◽

Sensing Data ◽

Effective Organization

Abstract. For the exploration and analysis of electricity, it is necessary to continuously acquire multi-star source, multi-temporal, multi-level remote sensing images for analysis and interpretation. Since the overall data has a variety of features, a data structure for multi-sensor data storage is proposed. On the basis of solving key technologies such as real-time image processing and analysis and remote sensing image normalization processing, the .xml file and remote sensing data geographic information file are used to realize effective organization between remote sensing data and remote sensing data. Based on GDAL design relational database, the formation of a relatively complete management system of data management, shared publishing and application services will maximize the potential value of remote sensing images in electricity remote sensing.

Download Full-text

Research on Improved Method of Storage and Query of Large-Scale Remote Sensing Images

Journal of Database Management ◽

10.4018/jdm.2018070101 ◽

2018 ◽

Vol 29 (3) ◽

pp. 1-16 ◽

Cited By ~ 1

Author(s):

Jing Weipeng ◽

Tian Dongxue ◽

Chen Guangsheng ◽

Li Yiyuan

Keyword(s):

Remote Sensing ◽

Data Storage ◽

Large Scale ◽

Remote Sensing Data ◽

Index Structure ◽

Hilbert Curve ◽

Remote Sensing Images ◽

Storage Model ◽

Sensing Data ◽

Low Efficiency

The traditional method is used to deal with massive remote sensing data stored in low efficiency and poor scalability. This article presents a parallel processing method based on MapReduce and HBase. The filling of remote sensing images by the Hilbert curve makes the MapReduce method construct pyramids in parallel to reduce network communication between nodes. Then, the authors design a massive remote sensing data storage model composed of metadata storage model, index structure and filter column family. Finally, this article uses MapReduce frameworks to realize pyramid construction, storage and query of remote sensing data. The experimental results show that this method can effectively improve the speed of data writing and querying, and has good scalability.

Download Full-text

Monitoring of a Grid Storage Virtualization Service

International Journal of Grid and High Performance Computing ◽

10.4018/jghpc.2013010104 ◽

2013 ◽

Vol 5 (1) ◽

pp. 53-69

Author(s):

Jacques Jorda ◽

Aurélien Ortiz ◽

Abdelaziz M’zoughi ◽

Salam Traboulsi

Keyword(s):

Monitoring System ◽

Data Storage ◽

Large Scale ◽

Distributed Storage ◽

Storage System ◽

Data Access ◽

Data Placement ◽

Workload Prediction ◽

Storage Virtualization

Grid computing is commonly used for large scale application requiring huge computation capabilities. In such distributed architectures, the data storage on the distributed storage resources must be handled by a dedicated storage system to ensure the required quality of service. In order to simplify the data placement on nodes and to increase the performance of applications, a storage virtualization layer can be used. This layer can be a single parallel filesystem (like GPFS) or a more complex middleware. The latter is preferred as it allows the data placement on the nodes to be tuned to increase both the reliability and the performance of data access. Thus, in such a middleware, a dedicated monitoring system must be used to ensure optimal performance. In this paper, the authors briefly introduce the Visage middleware – a middleware for storage virtualization. They present the most broadly used grid monitoring systems, and explain why they are not adequate for virtualized storage monitoring. The authors then present the architecture of their monitoring system dedicated to storage virtualization. We introduce the workload prediction model used to define the best node for data placement, and show on a simple experiment its accuracy.

Download Full-text

A Distributed Storage and Access Approach for Massive Remote Sensing Data in MongoDB

ISPRS International Journal of Geo-Information ◽

10.3390/ijgi8120533 ◽

2019 ◽

Vol 8 (12) ◽

pp. 533 ◽

Cited By ~ 3

Author(s):

Shuang Wang ◽

Guoqing Li ◽

Xiaochuang Yao ◽

Yi Zeng ◽

Lushen Pang ◽

...

Keyword(s):

Remote Sensing ◽

Relational Databases ◽

Large Scale ◽

Distributed Storage ◽

Rapid Development ◽

Remote Sensing Data ◽

Image Data ◽

Sensing Data ◽

Cluster Architecture ◽

Distributed Cluster

With the rapid development of earth-observation technology, the amount of remote sensing data has increased exponentially, and traditional relational databases cannot satisfy the requirements of managing large-scale remote sensing data. To address this problem, this paper undertakes intensive research of the NoSQL (Not Only SQL) data management model, especially the MongoDB database, and proposes a new approach to managing large-scale remote sensing data. Firstly, based on the sharding technology of MongoDB, a distributed cluster architecture was designed and established for massive remote sensing data. Secondly, for the convenience in the unified management of remote sensing data, an archiving model was constructed, and remote sensing data, including structured metadata and unstructured image data, were stored in the above cluster separately, with the metadata stored in the form of a document, and image data stored with the GridFS mechanism. Finally, by designing different shard strategies and comparing MongoDB cluster with a typical relational database, several groups of experiments were conducted to verify the storage performance and access performance of the cluster. The experimental results show that the proposed method can overcome the deficiencies of traditional methods, as well as scale out the database, which is more suitable for managing massive remote sensing data and can provide technical support for the management of massive remote sensing data.

Download Full-text

An Efficient Decision Support System for Flood Inundation Management Using Intermittent Remote-Sensing Data

Remote Sensing ◽

10.3390/rs13142818 ◽

2021 ◽

Vol 13 (14) ◽

pp. 2818

Author(s):

Hai Sun ◽

Xiaoyi Dai ◽

Wenchi Shou ◽

Jun Wang ◽

Xuejing Ruan

Keyword(s):

Remote Sensing ◽

Decision Support ◽

Decision Support System ◽

Support System ◽

Water Body ◽

Remote Sensing Data ◽

Flood Inundation ◽

Remote Sensing Images ◽

Sensing Data ◽

Tracking Model

Timely acquisition of spatial flood distribution is an essential basis for flood-disaster monitoring and management. Remote-sensing data have been widely used in water-body surveys. However, due to the cloudy weather and complex geomorphic environment, the inability to receive remote-sensing images throughout the day has resulted in some data being missing and unable to provide dynamic and continuous flood inundation process data. To fully and effectively use remote-sensing data, we developed a new decision support system for integrated flood inundation management based on limited and intermittent remote-sensing data. Firstly, we established a new multi-scale water-extraction convolutional neural network named DEU-Net to extract water from remote-sensing images automatically. A specific datasets training method was created for typical region types to separate the water body from the confusing surface features more accurately. Secondly, we built a waterfront contour active tracking model to implicitly describe the flood movement interface. In this way, the flooding process was converted into the numerical solution of the partial differential equation of the boundary function. Space upwind difference format and the time Euler difference format were used to perform the numerical solution. Finally, we established seven indicators that considered regional characteristics and flood-inundation attributes to evaluate flood-disaster losses. The cloud model using the entropy weight method was introduced to account for uncertainties in various parameters. In the end, a decision support system realizing the flood losses risk visualization was developed by using the ArcGIS application programming interface (API). To verify the effectiveness of the model constructed in this paper, we conducted numerical experiments on the model's performance through comparative experiments based on a laboratory scale and actual scale, respectively. The results were as follows: (1) The DEU-Net method had a better capability to accurately extract various water bodies, such as urban water bodies, open-air ponds, plateau lakes etc., than the other comparison methods. (2) The simulation results of the active tracking model had good temporal and spatial consistency with the image extraction results and actual statistical data compared with the synthetic observation data. (3) The application results showed that the system has high computational efficiency and noticeable visualization effects. The research results may provide a scientific basis for the emergency-response decision-making of flood disasters, especially in data-sparse regions.

Download Full-text

Conversion and Display of the Calculated Data of Spectral Remote Sensing Data on the Basis of GeoServer Extensions and Distributed Storage Technologies

PROGRAMMNAYA INGENERIA ◽

10.17587/prin.12.107-112 ◽

2021 ◽

Vol 12 (2) ◽

pp. 107-112

Author(s):

I. E. Kharlampenkov ◽

◽

A. U. Oshchepkov ◽

Keyword(s):

Remote Sensing ◽

Information Technologies ◽

File System ◽

Distributed Storage ◽

Calculated Data ◽

Remote Sensing Data ◽

Distributed Computing Systems ◽

Apache Hadoop ◽

Distributed Information ◽

Sensing Data

The article presents methods for caching and displaying data from spectral satellite images using libraries of distributed computing systems that are part of the Apache Hadoop ecosystem, and GeoServer extensions. The authors gave a brief overview of existing tools that provide the ability to present remote sensing data using distributed information technologies. A distinctive feature is the way to convert remote sensing data inside Apache Parquet files for further display. This approach allows you to interact with the distributed file system via the Kite SDK libraries and switch on additional data processors based on Apache Hadoop technology as external services. A comparative analysis of existing tools, such as: GeoMesa, GeoWawe, etc is performed. The following steps are described: extracting data from Apache Parquet via the Kite SDK, converting this data to GDAL Dataset, iterating the received data, and saving it inside the file system in BIL format. In this article, the BIL format is used for the GeoServer cache. The extension was implemented and published under the Apache License on the GitHub resource. In conclusion, you will find instructions for installing and using the created extension.

Download Full-text