A GPU-Accelerated In-Memory Metadata Management Scheme for Large-Scale Parallel File Systems

2021 ◽  
Vol 36 (1) ◽  
pp. 44-55
Author(s):  
Zhi-Guang Chen ◽  
Yu-Bo Liu ◽  
Yong-Feng Wang ◽  
Yu-Tong Lu
Keyword(s):  
Large Scale ◽  
File Systems ◽  
Metadata Management ◽  
Parallel File Systems ◽  
Management Scheme ◽  
Parallel File

DMFSsim: A Distributed Metadata File System Simulator

2012 ◽  
Vol 241-244 ◽  
pp. 1556-1561
Author(s):  
Qi Meng Wu ◽  
Ke Xie ◽  
Ming Fa Zhu ◽  
Li Min Xiao ◽  
Li Ruan
Keyword(s):  
Large Scale ◽  
File System ◽  
File Systems ◽  
Performance Gain ◽  
Parallel File Systems ◽  
Management Mechanism ◽  
Metadata File ◽  
Parallel File ◽  
Distribution Algorithms

Parallel file systems deploy multiple metadata servers to distribute heavy metadata workload from clients. With the increasing number of metadata servers, metadata-intensive operations are facing some problems related with collaboration among them, compromising the performance gain. Consequently, a file system simulator is very helpful to try out some optimization ideas to solve these problems. In this paper, we propose DMFSsim to simulate the metadata-intensive operations on large-scale distributed metadata file systems. DMFSsim can flexibly replay traces of multiple metadata operations, support several commonly used metadata distribution algorithms, simulate file system tree hierarchy and underlying disk blocks management mechanism in real systems. Extensive simulations show that DMFSsim is capable of demonstrating the performance of metadata-intensive operations in distributed metadata file system.

Rethinking key–value store for parallel I/O optimization

2016 ◽  
Vol 31 (4) ◽  
pp. 335-356 ◽  
Author(s):  
Anthony Kougkas ◽  
Hassan Eslami ◽  
Xian-He Sun ◽  
Rajeev Thakur ◽  
William Gropp
Keyword(s):  
Cloud Storage ◽  
Large Scale ◽  
Storage Systems ◽  
Storage System ◽  
File Systems ◽  
Data Synchronization ◽  
Input Output ◽  
Parallel File Systems ◽  
Parallel File ◽  
And Performance

Key–value stores are being widely used as the storage system for large-scale internet services and cloud storage systems. However, they are rarely used in HPC systems, where parallel file systems are the dominant storage solution. In this study, we examine the architecture differences and performance characteristics of parallel file systems and key–value stores. We propose using key–value stores to optimize overall Input/Output (I/O) performance, especially for workloads that parallel file systems cannot handle well, such as the cases with intense data synchronization or heavy metadata operations. We conducted experiments with several synthetic benchmarks, an I/O benchmark, and a real application. We modeled the performance of these two systems using collected data from our experiments, and we provide a predictive method to identify which system offers better I/O performance given a specific workload. The results show that we can optimize the I/O performance in HPC systems by utilizing key–value stores.

scholarly journals Flexibility and performance of parallel file systems

1996 ◽  
Vol 30 (2) ◽  
pp. 63-73 ◽  
Author(s):  
David Kotz ◽  
Nils Nieuwejaar
Keyword(s):  
File Systems ◽  
Parallel File Systems ◽  
Parallel File ◽  
And Performance
Sign in / Sign up

Export Citation Format

Share Document