scholarly journals DESIGN AND VERIFICATION OF REMOTE SENSING IMAGE DATA CENTER STORAGE ARCHITECTURE BASED ON HADOOP

2018 ◽  
Vol XLII-3 ◽  
pp. 1639-1642
Author(s):  
D. Tang ◽  
X. Zhou ◽  
Y. Jing ◽  
W. Cong ◽  
C. Li
Keyword(s):  
Remote Sensing ◽  
Data Storage ◽  
Data Center ◽  
High Speed ◽  
Large Scale ◽  
Distributed Storage ◽  
Image Data ◽  
Remotely Sensed Data ◽  
Processing Technologies ◽  
Data Parallel

The data center is a new concept of data processing and application proposed in recent years. It is a new method of processing technologies based on data, parallel computing, and compatibility with different hardware clusters. While optimizing the data storage management structure, it fully utilizes cluster resource computing nodes and improves the efficiency of data parallel application. This paper used mature Hadoop technology to build a large-scale distributed image management architecture for remote sensing imagery. Using MapReduce parallel processing technology, it called many computing nodes to process image storage blocks and pyramids in the background to improve the efficiency of image reading and application and sovled the need for concurrent multi-user high-speed access to remotely sensed data. It verified the rationality, reliability and superiority of the system design by testing the storage efficiency of different image data and multi-users and analyzing the distributed storage architecture to improve the application efficiency of remote sensing images through building an actual Hadoop service system.

scholarly journals A Distributed Storage and Access Approach for Massive Remote Sensing Data in MongoDB

2019 ◽  
Vol 8 (12) ◽  
pp. 533 ◽  
Author(s):  
Shuang Wang ◽  
Guoqing Li ◽  
Xiaochuang Yao ◽  
Yi Zeng ◽  
Lushen Pang ◽  
...  
Keyword(s):  
Remote Sensing ◽  
Relational Databases ◽  
Large Scale ◽  
Distributed Storage ◽  
Rapid Development ◽  
Remote Sensing Data ◽  
Image Data ◽  
Sensing Data ◽  
Cluster Architecture ◽  
Distributed Cluster

With the rapid development of earth-observation technology, the amount of remote sensing data has increased exponentially, and traditional relational databases cannot satisfy the requirements of managing large-scale remote sensing data. To address this problem, this paper undertakes intensive research of the NoSQL (Not Only SQL) data management model, especially the MongoDB database, and proposes a new approach to managing large-scale remote sensing data. Firstly, based on the sharding technology of MongoDB, a distributed cluster architecture was designed and established for massive remote sensing data. Secondly, for the convenience in the unified management of remote sensing data, an archiving model was constructed, and remote sensing data, including structured metadata and unstructured image data, were stored in the above cluster separately, with the metadata stored in the form of a document, and image data stored with the GridFS mechanism. Finally, by designing different shard strategies and comparing MongoDB cluster with a typical relational database, several groups of experiments were conducted to verify the storage performance and access performance of the cluster. The experimental results show that the proposed method can overcome the deficiencies of traditional methods, as well as scale out the database, which is more suitable for managing massive remote sensing data and can provide technical support for the management of massive remote sensing data.

Research on Distributed Heterogeneous Data Storage Algorithm in Cloud Computing Data Center

2014 ◽  
Vol 624 ◽  
pp. 553-556
Author(s):  
Jing Bo Yang ◽  
Shu Huang ◽  
Pan Jiang
Keyword(s):  
Cloud Computing ◽  
Data Storage ◽  
Data Center ◽  
Data Stream ◽  
Large Scale ◽  
Distributed Storage ◽  
Heterogeneous Data ◽  
Allocation Problem ◽  
Data Consistency ◽  
Storage Allocation

With the development of cloud computing, data center is also improved. cloud computing data center contains hundreds, even million of servers or PCs. It has many heterogeneous resources. Data center is a key to promise high scalability and resource usage of cloud computing. In addition, replica is introduced into data center, which is an important method to improve availability and performance. In this paper, the research on distributed storage algorithm based on the cloud computing. This algorithm uses the design of system storage level indicators within classification of massive data storage mechanism to solve the allocation problem of data consistency between the data center; and send communication packets between data centers through the cloud computing. The full storage can achieve complete local storage of each data stream, and solve the original data stream unusually large-scale data storage allocation problem.

scholarly journals RSIMS: Large-Scale Heterogeneous Remote Sensing Images Management System

2021 ◽  
Vol 13 (9) ◽  
pp. 1815
Author(s):  
Xiaohua Zhou ◽  
Xuezhi Wang ◽  
Yuanchun Zhou ◽  
Qinghui Lin ◽  
Jianghua Zhao ◽  
...  
Keyword(s):  
Remote Sensing ◽  
Data Storage ◽  
Management System ◽  
Large Scale ◽  
Distributed Storage ◽  
Storage System ◽  
Remote Sensing Data ◽  
Data Access ◽  
Remote Sensing Images ◽  
Sensing Data

With the remarkable development and progress of earth-observation techniques, remote sensing data keep growing rapidly and their volume has reached exabyte scale. However, it's still a big challenge to manage and process such huge amounts of remote sensing data with complex and diverse structures. This paper designs and realizes a distributed storage system for large-scale remote sensing data storage, access, and retrieval, called RSIMS (remote sensing images management system), which is composed of three sub-modules: RSIAPI, RSIMeta, RSIData. Structured text metadata of different remote sensing images are all stored in RSIMeta based on a set of uniform models, and then indexed by the distributed multi-level Hilbert grids for high spatiotemporal retrieval performance. Unstructured binary image files are stored in RSIData, which provides large scalable storage capacity and efficient GDAL (Geospatial Data Abstraction Library) compatible I/O interfaces. Popular GIS software and tools (e.g., QGIS, ArcGIS, rasterio) can access data stored in RSIData directly. RSIAPI provides users a set of uniform interfaces for data access and retrieval, hiding the complex inner structures of RSIMS. The test results show that RSIMS can store and manage large amounts of remote sensing images from various sources with high and stable performance, and is easy to deploy and use.

scholarly journals Multitemporal Mapping of Post-Fire Land Cover Using Multiplatform PRISMA Hyperspectral and Sentinel-UAV Multispectral Data: Insights from Case Studies in Portugal and Italy

Sensors ◽  
2021 ◽  
Vol 21 (12) ◽  
pp. 3982
Author(s):  
Giacomo Lazzeri ◽  
William Frodella ◽  
Guglielmo Rossi ◽  
Sandro Moretti
Keyword(s):  
Remote Sensing ◽  
Large Scale ◽  
Fire Severity ◽  
Experimental Methodology ◽  
Vegetation Recovery ◽  
Burned Area ◽  
Remotely Sensed Data ◽  
The Impact ◽  
Sentinel 2

Wildfires have affected global forests and the Mediterranean area with increasing recurrency and intensity in the last years, with climate change resulting in reduced precipitations and higher temperatures. To assess the impact of wildfires on the environment, burned area mapping has become progressively more relevant. Initially carried out via field sketches, the advent of satellite remote sensing opened new possibilities, reducing the cost uncertainty and safety of the previous techniques. In the present study an experimental methodology was adopted to test the potential of advanced remote sensing techniques such as multispectral Sentinel-2, PRISMA hyperspectral satellite, and UAV (unmanned aerial vehicle) remotely-sensed data for the multitemporal mapping of burned areas by soil–vegetation recovery analysis in two test sites in Portugal and Italy. In case study one, innovative multiplatform data classification was performed with the correlation between Sentinel-2 RBR (relativized burn ratio) fire severity classes and the scene hyperspectral signature, performed with a pixel-by-pixel comparison leading to a converging classification. In the adopted methodology, RBR burned area analysis and vegetation recovery was tested for accordance with biophysical vegetation parameters (LAI, fCover, and fAPAR). In case study two, a UAV-sensed NDVI index was adopted for high-resolution mapping data collection. At a large scale, the Sentinel-2 RBR index proved to be efficient for burned area analysis, from both fire severity and vegetation recovery phenomena perspectives. Despite the elapsed time between the event and the acquisition, PRISMA hyperspectral converging classification based on Sentinel-2 was able to detect and discriminate different spectral signatures corresponding to different fire severity classes. At a slope scale, the UAV platform proved to be an effective tool for mapping and characterizing the burned area, giving clear advantage with respect to filed GPS mapping. Results highlighted that UAV platforms, if equipped with a hyperspectral sensor and used in a synergistic approach with PRISMA, would create a useful tool for satellite acquired data scene classification, allowing for the acquisition of a ground truth.

scholarly journals Payload Configurations for Efficient Image Acquisition – Indian Perspective

2014 ◽  
Vol XL-8 ◽  
pp. 1201-1206 ◽  
Author(s):  
D. R. M. Samudraiah ◽  
M. Saxena ◽  
S. Paul ◽  
P. Narayanababu ◽  
S. Kuriakose ◽  
...  
Keyword(s):  
Remote Sensing ◽  
Data Acquisition ◽  
High Efficiency ◽  
Image Acquisition ◽  
Image Data ◽  
Space Research ◽  
Remotely Sensed Data ◽  
Data Utilization ◽  
Spectral Bands ◽  
The World

The world is increasingly depending on remotely sensed data. The data is regularly used for monitoring the earth resources and also for solving problems of the world like disasters, climate degradation, etc. Remotely sensed data has changed our perspective of understanding of other planets. With innovative approaches in data utilization, the demands of remote sensing data are ever increasing. More and more research and developments are taken up for data utilization. The satellite resources are scarce and each launch costs heavily. Each launch is also associated with large effort for developing the hardware prior to launch. It is also associated with large number of software elements and mathematical algorithms post-launch. The proliferation of low-earth and geostationary satellites has led to increased scarcity in the available orbital slots for the newer satellites. Indian Space Research Organization has always tried to maximize the utility of satellites. Multiple sensors are flown on each satellite. In each of the satellites, sensors are designed to cater to various spectral bands/frequencies, spatial and temporal resolutions. Bhaskara-1, the first experimental satellite started with 2 bands in electro-optical spectrum and 3 bands in microwave spectrum. The recent Resourcesat-2 incorporates very efficient image acquisition approach with multi-resolution (3 types of spatial resolution) multi-band (4 spectral bands) electro-optical sensors (LISS-4, LISS-3* and AWiFS). The system has been designed to provide data globally with various data reception stations and onboard data storage capabilities. Oceansat-2 satellite has unique sensor combination with 8 band electro-optical high sensitive ocean colour monitor (catering to ocean and land) along with Ku band scatterometer to acquire information on ocean winds. INSAT- 3D launched recently provides high resolution 6 band image data in visible, short-wave, mid-wave and long-wave infrared spectrum. It also has 19 band sounder for providing vertical profile of water vapour, temperature, etc. The same system has data relay transponders for acquiring data from weather stations. The payload configurations have gone through significant changes over the years to increase data rate per kilogram of payload. Future Indian remote sensing systems are planned with very high efficient ways of image acquisition. <br><br> This paper analyses the strides taken by ISRO (Indian Space research Organisation) in achieving high efficiency in remote sensing image data acquisition. Parameters related to efficiency of image data acquisition are defined and a methodology is worked out to compute the same. Some of the Indian payloads are analysed with respect to some of the system/ subsystem parameters that decide the configuration of payload. Based on the analysis, possible configuration approaches that can provide high efficiency are identified. A case study is carried out with improved configuration and the results of efficiency improvements are reported. This methodology may be used for assessing other electro-optical payloads or missions and can be extended to other types of payloads and missions.

scholarly journals RDCRMG: A Raster Dataset Clean & Reconstitution Multi-Grid Architecture for Remote Sensing Monitoring of Vegetation Dryness

2018 ◽  
Vol 10 (9) ◽  
pp. 1376 ◽  
Author(s):  
Sijing Ye ◽  
Diyou Liu ◽  
Xiaochuang Yao ◽  
Huaizhi Tang ◽  
Quan Xiong ◽  
...  
Keyword(s):  
Remote Sensing ◽  
Data Storage ◽  
Large Scale ◽  
High Efficiency ◽  
Data Extraction ◽  
Automatic Monitoring ◽  
Information Distortion ◽  
Remote Sensing Monitoring ◽  
Data Inversion ◽  
Analysis Models

In recent years, remote sensing (RS) research on crop growth status monitoring has gradually turned from static spectrum information retrieval in large-scale to meso-scale or micro-scale, timely multi-source data cooperative analysis; this change has presented higher requirements for RS data acquisition and analysis efficiency. How to implement rapid and stable massive RS data extraction and analysis becomes a serious problem. This paper reports on a Raster Dataset Clean & Reconstitution Multi-Grid (RDCRMG) architecture for remote sensing monitoring of vegetation dryness in which different types of raster datasets have been partitioned, organized and systematically applied. First, raster images have been subdivided into several independent blocks and distributed for storage in different data nodes by using the multi-grid as a consistent partition unit. Second, the “no metadata model” ideology has been referenced so that targets raster data can be speedily extracted by directly calculating the data storage path without retrieving metadata records; third, grids that cover the query range can be easily assessed. This assessment allows the query task to be easily split into several sub-tasks and executed in parallel by grouping these grids. Our RDCRMG-based change detection of the spectral reflectance information test and the data extraction efficiency comparative test shows that the RDCRMG is reliable for vegetation dryness monitoring with a slight reflectance information distortion and consistent percentage histograms. Furthermore, the RDCGMG-based data extraction in parallel circumstances has the advantages of high efficiency and excellent stability compared to that of the RDCGMG-based data extraction in serial circumstances and traditional data extraction. At last, an RDCRMG-based vegetation dryness monitoring platform (VDMP) has been constructed to apply RS data inversion in vegetation dryness monitoring. Through actual applications, the RDCRMG architecture is proven to be appropriate for timely vegetation dryness RS automatic monitoring with better performance, more reliability and higher extensibility. Our future works will focus on integrating more kinds of continuously updated RS data into the RDCRMG-based VDMP and integrating more multi-source datasets based collaborative analysis models for agricultural monitoring.

Monitoring of a Grid Storage Virtualization Service

2013 ◽  
Vol 5 (1) ◽  
pp. 53-69
Author(s):  
Jacques Jorda ◽  
Aurélien Ortiz ◽  
Abdelaziz M’zoughi ◽  
Salam Traboulsi
Keyword(s):  
Monitoring System ◽  
Data Storage ◽  
Large Scale ◽  
Distributed Storage ◽  
Storage System ◽  
Data Access ◽  
Data Placement ◽  
Workload Prediction ◽  
Storage Virtualization

Grid computing is commonly used for large scale application requiring huge computation capabilities. In such distributed architectures, the data storage on the distributed storage resources must be handled by a dedicated storage system to ensure the required quality of service. In order to simplify the data placement on nodes and to increase the performance of applications, a storage virtualization layer can be used. This layer can be a single parallel filesystem (like GPFS) or a more complex middleware. The latter is preferred as it allows the data placement on the nodes to be tuned to increase both the reliability and the performance of data access. Thus, in such a middleware, a dedicated monitoring system must be used to ensure optimal performance. In this paper, the authors briefly introduce the Visage middleware – a middleware for storage virtualization. They present the most broadly used grid monitoring systems, and explain why they are not adequate for virtualized storage monitoring. The authors then present the architecture of their monitoring system dedicated to storage virtualization. We introduce the workload prediction model used to define the best node for data placement, and show on a simple experiment its accuracy.

Image Data Hard Disk High-Speed Storage Technology of Visible/Infrared Dual-Band Aerial Remote Sensing Camera

2013 ◽  
Vol 28 (6) ◽  
pp. 895-900
Author(s):  
韩松伟 HAN Song-wei ◽  
孙丽娜 SUN Li-na ◽  
孟中 MENG Zhong ◽  
毛大鹏 MAO Da-peng ◽  
张建华 ZHANG Jian-hua
Keyword(s):  
Remote Sensing ◽  
High Speed ◽  
Image Data ◽  
Hard Disk ◽  
Dual Band ◽  
Storage Technology ◽  
Aerial Remote Sensing

A Review: Map Reduce Framework for Cloud Computing

2018 ◽  
Vol 7 (4.6) ◽  
pp. 13
Author(s):  
Mekala Sandhya ◽  
Ashish Ladda ◽  
Dr. Uma N Dulhare ◽  
. . ◽  
. .
Keyword(s):  
Data Mining ◽  
Cloud Computing ◽  
Distributed Computing ◽  
Data Storage ◽  
High Performance ◽  
Large Scale ◽  
Distributed Storage ◽  
Large Data ◽  
Mass Data ◽  
Internet Information

In this generation of Internet, information and data are growing continuously. Even though various Internet services and applications. The amount of information is increasing rapidly. Hundred billions even trillions of web indexes exist. Such large data brings people a mass of information and more difficulty discovering useful knowledge in these huge amounts of data at the same time. Cloud computing can provide infrastructure for large data. Cloud computing has two significant characteristics of distributed computing i.e. scalability, high availability. The scalability can seamlessly extend to large-scale clusters. Availability says that cloud computing can bear node errors. Node failures will not affect the program to run correctly. Cloud computing with data mining does significant data processing through high-performance machine. Mass data storage and distributed computing provide a new method for mass data mining and become an effective solution to the distributed storage and efficient computing in data mining. 

Geostatistics and remote sensing

1998 ◽  
Vol 22 (1) ◽  
pp. 61-78 ◽  
Author(s):  
Paul J. Curran ◽  
Peter M. Atkinson
Keyword(s):  
Remote Sensing ◽  
Spatial Variation ◽  
Conditional Simulation ◽  
Image Data ◽  
Remotely Sensed ◽  
Continuous Variables ◽  
Remotely Sensed Data ◽  
Sampling Schemes ◽  
Optimum Sampling ◽  
Design Optimum

In geostatistics, spatial autocorrelation is utilized to estimate optimally local values from data sampled elsewhere. The powerful synergy between geostatistics and remote sensing went unrealized until the 1980s. Today geostatistics are used to explore and describe spatial variation in remotely sensed and ground data; to design optimum sampling schemes for image data and ground data; and to increase the accuracy with which remotely sensed data can be used to classify land cover or estimate continuous variables. This article introduces these applications and uses two examples to highlight characteristics that are common to them all. The article concludes with a discussion of conditional simulation as a novel geostatistical technique for use in remote sensing.

Sign in / Sign up

Export Citation Format

Share Document