A Distributed Storage and Access Approach for Massive Remote Sensing Data in MongoDB

With the rapid development of earth-observation technology, the amount of remote sensing data has increased exponentially, and traditional relational databases cannot satisfy the requirements of managing large-scale remote sensing data. To address this problem, this paper undertakes intensive research of the NoSQL (Not Only SQL) data management model, especially the MongoDB database, and proposes a new approach to managing large-scale remote sensing data. Firstly, based on the sharding technology of MongoDB, a distributed cluster architecture was designed and established for massive remote sensing data. Secondly, for the convenience in the unified management of remote sensing data, an archiving model was constructed, and remote sensing data, including structured metadata and unstructured image data, were stored in the above cluster separately, with the metadata stored in the form of a document, and image data stored with the GridFS mechanism. Finally, by designing different shard strategies and comparing MongoDB cluster with a typical relational database, several groups of experiments were conducted to verify the storage performance and access performance of the cluster. The experimental results show that the proposed method can overcome the deficiencies of traditional methods, as well as scale out the database, which is more suitable for managing massive remote sensing data and can provide technical support for the management of massive remote sensing data.

Download Full-text

RSIMS: Large-Scale Heterogeneous Remote Sensing Images Management System

Remote Sensing ◽

10.3390/rs13091815 ◽

2021 ◽

Vol 13 (9) ◽

pp. 1815

Author(s):

Xiaohua Zhou ◽

Xuezhi Wang ◽

Yuanchun Zhou ◽

Qinghui Lin ◽

Jianghua Zhao ◽

...

Keyword(s):

Remote Sensing ◽

Data Storage ◽

Management System ◽

Large Scale ◽

Distributed Storage ◽

Storage System ◽

Remote Sensing Data ◽

Data Access ◽

Remote Sensing Images ◽

Sensing Data

With the remarkable development and progress of earth-observation techniques, remote sensing data keep growing rapidly and their volume has reached exabyte scale. However, it's still a big challenge to manage and process such huge amounts of remote sensing data with complex and diverse structures. This paper designs and realizes a distributed storage system for large-scale remote sensing data storage, access, and retrieval, called RSIMS (remote sensing images management system), which is composed of three sub-modules: RSIAPI, RSIMeta, RSIData. Structured text metadata of different remote sensing images are all stored in RSIMeta based on a set of uniform models, and then indexed by the distributed multi-level Hilbert grids for high spatiotemporal retrieval performance. Unstructured binary image files are stored in RSIData, which provides large scalable storage capacity and efficient GDAL (Geospatial Data Abstraction Library) compatible I/O interfaces. Popular GIS software and tools (e.g., QGIS, ArcGIS, rasterio) can access data stored in RSIData directly. RSIAPI provides users a set of uniform interfaces for data access and retrieval, hiding the complex inner structures of RSIMS. The test results show that RSIMS can store and manage large amounts of remote sensing images from various sources with high and stable performance, and is easy to deploy and use.

Download Full-text

AN APPROACH FOR EVALUATING THE INFORMATION CONTENT OF REMOTE SENSING IMAGES

ISPRS - International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences ◽

10.5194/isprs-archives-xlii-4-w20-21-2019 ◽

2019 ◽

Vol XLII-4/W20 ◽

pp. 21-26

Author(s):

S. M. Fang ◽

X. G. Zhou

Keyword(s):

Remote Sensing ◽

Information Content ◽

Evaluation Method ◽

Rapid Development ◽

Remote Sensing Data ◽

Image Data ◽

Structural Features ◽

Utilization Efficiency ◽

Sensing Data

Abstract. Due to being affected by the rapid development of open science and the increasing popularity of mobile devices (e.g., smartphones), remote sensing data as frequently used data sources are broadly applied to our daily life. At the same time, remote sensing data collection also presents a trend of popularization. To improve the utilization efficiency and availability of the obtained diversified remote sensing data, we propose a novel evaluation method based on information theory and scatterplot mapping model, i.e., geometrical mapping entropy (GME). The goal is to construct a unified model of measurement to be much more effectively and accurately evaluate the information content and quality of remotely sensed imagery. Different experimental data are used to verify the performance of the proposed method, i.e., a group of the dataset that contains different four types of images; the other group of image data contains the images with different modalities and different imaging times (2016–05, 2017–08, 2018–04, and 2018–06). Experimental results indicate that the proposed approach can better characterize the spectrum features and spatial structural features contained in images and visual perception information. Additionally, it can also reflect the difference in the quality of different modality images, especially the effect for the images that contain clouds or poor lighting conditions, is better.

Download Full-text

Conversion and Display of the Calculated Data of Spectral Remote Sensing Data on the Basis of GeoServer Extensions and Distributed Storage Technologies

PROGRAMMNAYA INGENERIA ◽

10.17587/prin.12.107-112 ◽

2021 ◽

Vol 12 (2) ◽

pp. 107-112

Author(s):

I. E. Kharlampenkov ◽

◽

A. U. Oshchepkov ◽

Keyword(s):

Remote Sensing ◽

Information Technologies ◽

File System ◽

Distributed Storage ◽

Calculated Data ◽

Remote Sensing Data ◽

Distributed Computing Systems ◽

Apache Hadoop ◽

Distributed Information ◽

Sensing Data

The article presents methods for caching and displaying data from spectral satellite images using libraries of distributed computing systems that are part of the Apache Hadoop ecosystem, and GeoServer extensions. The authors gave a brief overview of existing tools that provide the ability to present remote sensing data using distributed information technologies. A distinctive feature is the way to convert remote sensing data inside Apache Parquet files for further display. This approach allows you to interact with the distributed file system via the Kite SDK libraries and switch on additional data processors based on Apache Hadoop technology as external services. A comparative analysis of existing tools, such as: GeoMesa, GeoWawe, etc is performed. The following steps are described: extracting data from Apache Parquet via the Kite SDK, converting this data to GDAL Dataset, iterating the received data, and saving it inside the file system in BIL format. In this article, the BIL format is used for the GeoServer cache. The extension was implemented and published under the Apache License on the GitHub resource. In conclusion, you will find instructions for installing and using the created extension.

Download Full-text

LESS: LargE-Scale remote sensing data and image simulation framework over heterogeneous 3D scenes

Remote Sensing of Environment ◽

10.1016/j.rse.2018.11.036 ◽

2019 ◽

Vol 221 ◽

pp. 695-706 ◽

Cited By ~ 12

Author(s):

Jianbo Qi ◽

Donghui Xie ◽

Tiangang Yin ◽

Guangjian Yan ◽

Jean-Philippe Gastellu-Etchegorry ◽

...

Keyword(s):

Remote Sensing ◽

Large Scale ◽

Remote Sensing Data ◽

Image Simulation ◽

Simulation Framework ◽

Sensing Data

Download Full-text

An efficient organization method for large-scale and long time-series remote sensing data in a cloud computing environment

IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing ◽

10.1109/jstars.2021.3110900 ◽

2021 ◽

pp. 1-1

Author(s):

Jining Yan ◽

Yuanxing Liu ◽

Lizhe Wang ◽

Zhipeng Wang ◽

Xiaohui Huang ◽

...

Keyword(s):

Remote Sensing ◽

Time Series ◽

Cloud Computing ◽

Large Scale ◽

Remote Sensing Data ◽

Computing Environment ◽

Cloud Computing Environment ◽

Efficient Organization ◽

Sensing Data ◽

Long Time

Download Full-text

Toward a Standardized Encoding of Remote Sensing Geo-Positioning Sensor Models

Remote Sensing ◽

10.3390/rs12091530 ◽

2020 ◽

Vol 12 (9) ◽

pp. 1530

Author(s):

Meng Jin ◽

Yuqi Bai ◽

Emmanuel Devys ◽

Liping Di

Keyword(s):

Remote Sensing ◽

Remote Sensing Data ◽

Image Data ◽

Xml Schema ◽

Data Interoperability ◽

Sensing Data ◽

Sensor Model ◽

Application Systems ◽

Extensible Markup ◽

Sensor Models

Geolocation information is an important feature of remote sensing image data that is captured through a variety of passive or active observation sensors, such as push-broom electro-optical sensor, synthetic aperture radar (SAR), light detection and ranging (LIDAR) and sound navigation and ranging (SONAR). As a fundamental processing step to locate an image, geo-positioning is used to determine the ground coordinates of an object from image coordinates. A variety of sensor models have been created to describe geo-positioning process. In particular, Open Geospatial Consortium (OGC) has defined the Sensor Model Language (SensorML) specification in its Sensor Web Enablement (SWE) initiative to describe sensors including the geo-positioning process. It has been realized using syntax from the extensible markup language (XML). Besides, two standards defined by the International Organization for Standardization (ISO), ISO 19130-1 and ISO 19130-2, introduced a physical sensor model, a true replacement model, and a correspondence model for the geo-positioning process. However, a standardized encoding for geo-positioning sensor models is still missing for the remote sensing community. Thus, the interoperability of remote sensing data between application systems cannot be ensured. In this paper, a standardized encoding of remote sensing geo-positioning sensor models is introduced. It is semantically based on ISO 19130-1 and ISO 19130-2, and syntactically based on OGC SensorML. It defines a cross mapping of the sensor models defined in ISO 19130-1 and ISO 19130-2 to the SensorML, and then proposes a detailed encoding method to finalize the XML schema (an XML schema here is the structure to define an XML document), which will become a profile of OGC SensorML. It seamlessly unifies the sensor models defined in ISO 19130-1, ISO 19130-2, and OGC SensorML. By enabling a standardized description of sensor models used to produce remote sensing data, this standard is very promising in promoting data interoperability, mobility, and integration in the remote sensing domain.

Download Full-text