Research on the Metadata Storage Mode and Efficiency of Distributed File System Based on HGML

2010 ◽  
Vol 40-41 ◽  
pp. 221-227 ◽  
Author(s):  
Fang Miao ◽  
Fu Chao Cheng ◽  
Wen Hui Yang ◽  
Li Tan
Keyword(s):  
Spatial Data ◽  
File System ◽  
Distributed File System ◽  
Entire System ◽  
The Core ◽  
Different Characteristics ◽  
Massive Spatial Data ◽  
Access Efficiency

In the G / S mode, in order to meet the storage demands of massive spatial data, the requirements of the distributed file system (DFS) on back-end servers are extremely high. As one of the core tasks of DFS, the metadata storage is the necessary premise which ensures the reliability and efficiency of the entire system. This paper introduces a metadata storage mode based on HGML, and then designs and implements two solutions, which are scattered storage and integrated storage. According to the different characteristics of the two solutions, access efficiency of the metadata has been tested respectively. The result shows that the new metadata storage mode can basically satisfy the storage demands of massive spatial data.

scholarly journals Rasterhadoop: An Application Perspective of Raster Data Processing on Hadoop

2019 ◽  
Vol 8 (4) ◽  
pp. 11147-11150
Keyword(s):  
Remote Sensing ◽  
Image Processing ◽  
Parallel Processing ◽  
Data Processing ◽  
Satellite Data ◽  
Spatial Data ◽  
File System ◽  
Distributed File System ◽  
Raster Data ◽  
Volume Velocity

Hadoop is currently the most popular platform for parallel processing. With its two major components namely the Distributed File System (HDFS) and a parallel processing paradigm (MapReduce) in addition to its ease of installation and usage, Hadoop has become the chosen platform for efficiency whether in the commercial arena or the scientific arena such as Satellite Data Processing. The number of remote sensing satellites have also grown leaps and bounds and the data sent back by them for processing has all the three characteristics namely volume, velocity and variety that make it Big Spatial Data. In this paper, we present the extensions provided to Hadoop that enable Image Processing using legacy code and further elaborate on the various methods provided.

Energy-Efficient Algorithms for Distributed File System HDFS

2014 ◽  
Vol 36 (5) ◽  
pp. 1047-1064 ◽  
Author(s):  
Bin LIAO ◽  
Jiong YU ◽  
Tao ZHANG ◽  
Xing-Yao YANG
Keyword(s):  
Energy Efficient ◽  
File System ◽  
Efficient Algorithms ◽  
Distributed File System

Performance Testing and Analysis to Storage Medium of Distributed File System

2010 ◽  
Vol 33 (10) ◽  
pp. 1873-1880 ◽  
Author(s):  
Chun-Cong XU ◽  
Xiao-Meng HUANG ◽  
Nuo WU ◽  
Ning-Wei SUN ◽  
Guang-Wen YANG
Keyword(s):  
File System ◽  
Performance Testing ◽  
Distributed File System ◽  
Storage Medium

Improving downloading performance in hadoop distributed file system

2010 ◽  
Vol 30 (8) ◽  
pp. 2060-2065 ◽  
Author(s):  
Ning CAO ◽  
Zhong-hai WU ◽  
Hong-zhi LIU ◽  
Qi-xun ZHANG
Keyword(s):  
File System ◽  
Distributed File System ◽  
Hadoop Distributed File System

scholarly journals Lustre I/O performance investigations on Hazel Hen: experiments and heuristics

2021 ◽  
Author(s):  
Marco Seiz ◽  
Philipp Offenhäuser ◽  
Stefan Andersson ◽  
Johannes Hötzer ◽  
Henrik Hierl ◽  
...  
Keyword(s):  
File System ◽  
Block Size ◽  
Significant Part ◽  
Computational Power ◽  
Test Program ◽  
The Core ◽  
Data Output ◽  
Small Test ◽  
User Space

AbstractWith ever-increasing computational power, larger computational domains are employed and thus the data output grows as well. Writing this data to disk can become a significant part of runtime if done serially. Even if the output is done in parallel, e.g., via MPI I/O, there are many user-space parameters for tuning the performance. This paper focuses on the available parameters for the Lustre file system and the Cray MPICH implementation of MPI I/O. Experiments on the Cray XC40 Hazel Hen using a Cray Sonexion 2000 Lustre file system were conducted. In the experiments, the core count, the block size and the striping configuration were varied. Based on these parameters, heuristics for striping configuration in terms of core count and block size were determined, yielding up to a 32-fold improvement in write rate compared to the default. This corresponds to 85 GB/s of the peak bandwidth of 202.5 GB/s. The heuristics are shown to be applicable to a small test program as well as a complex application.

scholarly journals Data protection on hadoop distributed file system by using encryption algorithms: a systematic literature review

2020 ◽  
Vol 1444 ◽  
pp. 012012
Author(s):  
Meisuchi Naisuty ◽  
Achmad Nizar Hidayanto ◽  
Nabila Clydea Harahap ◽  
Ahmad Rosyiq ◽  
Agus Suhanto ◽  
...  
Keyword(s):  
Literature Review ◽  
Systematic Literature Review ◽  
Data Protection ◽  
File System ◽  
Distributed File System ◽  
Hadoop Distributed File System ◽  
Encryption Algorithms
Sign in / Sign up

Export Citation Format

Share Document