High Performance Metadata Management Engine for Large-Scale Distributed File Systems

2015 ◽  
Author(s):  
Myung-Hoon Cha ◽  
Sang-Min Lee ◽  
Dong-Oh Kim ◽  
Hong-Yeon Kim ◽  
Young-Kyun Kim
Keyword(s):  
High Performance ◽  
Large Scale ◽  
File Systems ◽  
Metadata Management ◽  
Distributed File Systems

An Efficient Ring-Based Metadata Management Policy for Large-Scale Distributed File Systems

2019 ◽  
Vol 30 (9) ◽  
pp. 1962-1974 ◽  
Author(s):  
Yuanning Gao ◽  
Xiaofeng Gao ◽  
Xiaochun Yang ◽  
Jiaxi Liu ◽  
Guihai Chen
Keyword(s):  
Large Scale ◽  
File Systems ◽  
Management Policy ◽  
Metadata Management ◽  
Distributed File Systems

Octopus + : An RDMA-Enabled Distributed Persistent Memory File System

2021 ◽  
Vol 17 (3) ◽  
pp. 1-25
Author(s):  
Bohong Zhu ◽  
Youmin Chen ◽  
Qing Wang ◽  
Youyou Lu ◽  
Jiwu Shu
Keyword(s):  
High Speed ◽  
High Performance ◽  
File System ◽  
Direct Memory Access ◽  
File Systems ◽  
Distributed File Systems ◽  
Persistent Memory ◽  
Memory Modules ◽  
Non Volatile Memory ◽  
Volatile Memory

Non-volatile memory and remote direct memory access (RDMA) provide extremely high performance in storage and network hardware. However, existing distributed file systems strictly isolate file system and network layers, and the heavy layered software designs leave high-speed hardware under-exploited. In this article, we propose an RDMA-enabled distributed persistent memory file system, Octopus + , to redesign file system internal mechanisms by closely coupling non-volatile memory and RDMA features. For data operations, Octopus + directly accesses a shared persistent memory pool to reduce memory copying overhead, and actively fetches and pushes data all in clients to rebalance the load between the server and network. For metadata operations, Octopus + introduces self-identified remote procedure calls for immediate notification between file systems and networking, and an efficient distributed transaction mechanism for consistency. Octopus + is enabled with replication feature to provide better availability. Evaluations on Intel Optane DC Persistent Memory Modules show that Octopus + achieves nearly the raw bandwidth for large I/Os and orders of magnitude better performance than existing distributed file systems.

Optimizing of metadata management in large-scale file systems

2018 ◽  
Vol 21 (4) ◽  
pp. 1865-1879 ◽  
Author(s):  
Nae Young Song ◽  
Hwajung Kim ◽  
Hyuck Han ◽  
Heon Young Yeom
Keyword(s):  
Large Scale ◽  
File Systems ◽  
Metadata Management

scholarly journals Scalable and Adaptive Metadata Management in Ultra Large-Scale File Systems

2008 ◽  
Author(s):  
Yu Hua ◽  
Yifeng Zhu ◽  
Hong Jiang ◽  
Dan Feng ◽  
Lei Tian
Keyword(s):  
Large Scale ◽  
File Systems ◽  
Metadata Management

scholarly journals Parallel network file systems using authenticated key exchange protocols

2017 ◽  
Vol 2 (3) ◽  
pp. 161
Author(s):  
S. Sathya ◽  
M. Ranjith Kumar ◽  
K. Madheswaran
Keyword(s):  
Large Scale ◽  
File Systems ◽  
Key Exchange ◽  
Distributed File Systems ◽  
Authenticated Key Exchange ◽  
Forward Secrecy ◽  
Storage Devices ◽  
Key Exchange Protocols ◽  
Parallel Network ◽  
Metadata Server

The keyestablishment for secure many-to-many communications is very important nowadays. The problem is inspired by the proliferation of large-scale distributed file systems supporting parallel access to multiple storage devices. In this, a variety of authenticated key exchange protocols that are designed to address the issues. This shows that these protocols are capable of reducing the workload of the metadata server and concurrently supporting forward secrecy and escrow-freeness. All this requires only a small fraction of increased computation overhead at the client. This proposed three authenticated key exchange protocols for parallel network file system (pNFS). The protocols offer three appealing advantages over the existing Kerberos-based protocol. First, the metadata server executing these protocols has much lower workload than that of the Kerberos-based approach. Second, two of these protocols provide forward secrecy: one is partially forward secure (with respect to multiple sessions within a time period), while the other is fully forward secure (with respect to a session). Third, designed a protocol which not only provides forward secrecy, but is also escrow-free.

scholarly journals Modeling of distributed file System in big data storage by event- B

2018 ◽  
Vol 210 ◽  
pp. 04042
Author(s):  
Ammar Alhaj Ali ◽  
Pavel Varacha ◽  
Said Krayem ◽  
Roman Jasek ◽  
Petr Zacek ◽  
...  
Keyword(s):  
Big Data ◽  
Data Storage ◽  
High Performance ◽  
File System ◽  
Formal Method ◽  
File Systems ◽  
Distributed File System ◽  
Distributed File Systems ◽  
Data Systems ◽  
Big Data Systems

Nowadays, a wide set of systems and application, especially in high performance computing, depends on distributed environments to process and analyses huge amounts of data. As we know, the amount of data increases enormously, and the goal to provide and develop efficient, scalable and reliable storage solutions has become one of the major issue for scientific computing. The storage solution used by big data systems is Distributed File Systems (DFSs), where DFS is used to build a hierarchical and unified view of multiple file servers and shares on the network. In this paper we will offer Hadoop Distributed File System (HDFS) as DFS in big data systems and we will present an Event-B as formal method that can be used in modeling, where Event-B is a mature formal method which has been widely used in a number of industry projects in a number of domains, such as automotive, transportation, space, business information, medical device and so on, And will propose using the Rodin as modeling tool for Event-B, which integrates modeling and proving as well as the Rodin platform is open source, so it supports a large number of plug-in tools.

scholarly journals Performance evaluation of a throughput-aware framework for ensemble data assimilation: the case of NICAM-LETKF

2016 ◽  
Vol 9 (7) ◽  
pp. 2293-2300 ◽  
Author(s):  
Hisashi Yashiro ◽  
Koji Terasaki ◽  
Takemasa Miyoshi ◽  
Hirofumi Tomita
Keyword(s):  
Data Assimilation ◽  
High Performance ◽  
Weather Prediction ◽  
File Systems ◽  
Atmospheric Model ◽  
Distributed File Systems ◽  
Ensemble Data Assimilation ◽  
Computational Performance ◽  
Ensemble Data ◽  
Data Throughput

Abstract. In this paper, we propose the design and implementation of an ensemble data assimilation (DA) framework for weather prediction at a high resolution and with a large ensemble size. We consider the deployment of this framework on the data throughput of file input/output (I/O) and multi-node communication. As an instance of the application of the proposed framework, a local ensemble transform Kalman filter (LETKF) was used with a Non-hydrostatic Icosahedral Atmospheric Model (NICAM) for the DA system. Benchmark tests were performed using the K computer, a massive parallel supercomputer with distributed file systems. The results showed an improvement in total time required for the workflow as well as satisfactory scalability of up to 10 K nodes (80 K cores). With regard to high-performance computing systems, where data throughput performance increases at a slower rate than computational performance, our new framework for ensemble DA systems promises drastic reduction of total execution time.

Sign in / Sign up

Export Citation Format

Share Document