Research on Distributed Storage Method for Big Data of Geological and Mineral Resources Based on Hadoop

The arrival of the era of big data for Earth observation (EO) indicates that traditional data management models have been unable to meet the needs of remote sensing data in big data environments. With the launch of the first remote sensing satellite, the volume of remote sensing data has also been increasing, and traditional data storage methods have been unable to ensure the efficient management of large amounts of remote sensing data. Therefore, a professional remote sensing big data integration method is sorely needed. In recent years, the emergence of some new technical methods has provided effective solutions for multi-source remote sensing data integration. This paper proposes a multi-source remote sensing data integration framework based on a distributed management model. In this framework, the multi-source remote sensing data are partitioned by the proposed spatial segmentation indexing (SSI) model through spatial grid segmentation. The designed complete information description system, based on International Organization for Standardization (ISO) 19115, can explain multi-source remote sensing data in detail. Then, the distributed storage method of data based on MongoDB is used to store multi-source remote sensing data. The distributed storage method is physically based on the sharding mechanism of the MongoDB database, and it can provide advantages for the security and performance of the preservation of remote sensing data. Finally, several experiments have been designed to test the performance of this framework in integrating multi-source remote sensing data. The results show that the storage and retrieval performance of the distributed remote sensing data integration framework proposed in this paper is superior. At the same time, the grid level of the SSI model proposed in this paper also has an important impact on the storage efficiency of remote sensing data. Therefore, the remote storage data integration framework, based on distributed storage, can provide new technical support and development prospects for big EO data.

Download Full-text

A Distributed Storage Method for High Speed Data Transmission System

2020 Cross Strait Radio Science & Wireless Technology Conference (CSRSWTC) ◽

10.1109/csrswtc50769.2020.9372702 ◽

2020 ◽

Author(s):

Jie Li ◽

Ruiliang Song

Keyword(s):

Data Transmission ◽

High Speed ◽

Distributed Storage ◽

Transmission System ◽

Storage Method ◽

High Speed Data ◽

Data Transmission System ◽

Speed Data Transmission

Download Full-text

Geospatial Serverless Computing: Architectures, Tools and Future Directions

ISPRS International Journal of Geo-Information ◽

10.3390/ijgi9050311 ◽

2020 ◽

Vol 9 (5) ◽

pp. 311 ◽

Cited By ~ 1

Author(s):

Sujit Bebortta ◽

Saneev Kumar Das ◽

Meenakshi Kandpal ◽

Rabindra Kumar Barik ◽

Harishchandra Dubey

Keyword(s):

Big Data ◽

Forest Cover ◽

Mineral Resources ◽

Hyperspectral Data ◽

Real Time Analysis ◽

First Case ◽

Amazon Web Services ◽

Aws Lambda ◽

Physical Features

Several real-world applications involve the aggregation of physical features corresponding to different geographic and topographic phenomena. This information plays a crucial role in analyzing and predicting several events. The application areas, which often require a real-time analysis, include traffic flow, forest cover, disease monitoring and so on. Thus, most of the existing systems portray some limitations at various levels of processing and implementation. Some of the most commonly observed factors involve lack of reliability, scalability and exceeding computational costs. In this paper, we address different well-known scalable serverless frameworks i.e., Amazon Web Services (AWS) Lambda, Google Cloud Functions and Microsoft Azure Functions for the management of geospatial big data. We discuss some of the existing approaches that are popularly used in analyzing geospatial big data and indicate their limitations. We report the applicability of our proposed framework in context of Cloud Geographic Information System (GIS) platform. An account of some state-of-the-art technologies and tools relevant to our problem domain are discussed. We also visualize performance of the proposed framework in terms of reliability, scalability, speed and security parameters. Furthermore, we present the map overlay analysis, point-cluster analysis, the generated heatmap and clustering analysis. Some relevant statistical plots are also visualized. In this paper, we consider two application case-studies. The first case study was explored using the Mineral Resources Data System (MRDS) dataset, which refers to worldwide density of mineral resources in a country-wise fashion. The second case study was performed using the Fairfax Forecast Households dataset, which signifies the parcel-level household prediction for 30 consecutive years. The proposed model integrates a serverless framework to reduce timing constraints and it also improves the performance associated to geospatial data processing for high-dimensional hyperspectral data.

Download Full-text

Storage Solution: A Virtual Distributed Storage And Migration Architecture For Big Data

Proceedings of the 2017 2nd Joint International Information Technology, Mechanical and Electronic Engineering Conference (JIMEC 2017) ◽

10.2991/jimec-17.2017.57 ◽

2017 ◽

Author(s):

Randle Oluwarotimi ◽

Matsebula Fezile ◽

Zuva Tranos

Keyword(s):

Big Data ◽

Distributed Storage ◽

And Migration ◽

Storage Solution

Download Full-text

Research on Environmental Monitoring System Based on Microservices and Data Mining

E3S Web of Conferences ◽

10.1051/e3sconf/202014302031 ◽

2020 ◽

Vol 143 ◽

pp. 02031

Author(s):

Yu Liu ◽

Junge Huang ◽

Ningqi Lu

Keyword(s):

Big Data ◽

Monitoring System ◽

Distributed Storage ◽

Automatic Monitoring ◽

Selective Extraction ◽

Monitoring Data ◽

Environment Monitoring ◽

Business Systems ◽

Efficient System ◽

Environmental Monitoring System

The monitoring systems of various industries have various types and different structures. There are problems of “data chimney” and “information islands”[1]. Monitoring data is difficult to be effectively utilized and cannot provide reliable data information to support for environmental security[2]. In this end, an environment monitoring system based on micro-service architecture is designed. The information management and automatic monitoring business systems are unified into a flexible, robust and efficient system platform to adapt to the big data analysis and the mining applications. Using Hadoop to build environment monitoring big data platform, distributed storage, selective extraction and efficient calculation of the massive environment monitoring data can be achieved. By integrating the detection and monitoring data of the ecological environment and in-depth mining it, a neural network model is established to automatically identify potential safety hazards and recommend corresponding treatment measures, so to assist in the comprehensive research and scientific decision-making of environmental safety and promote intelligent management of safety.

Download Full-text