Striping Layout Aware Data Aggregation for High Performance I/O on a Lustre File System

2015 ◽  
pp. 282-290 ◽  
Author(s):  
Yuichi Tsujita ◽  
Atsushi Hori ◽  
Yutaka Ishikawa
Keyword(s):  
Data Aggregation ◽  
High Performance ◽  
File System

Octopus + : An RDMA-Enabled Distributed Persistent Memory File System

2021 ◽  
Vol 17 (3) ◽  
pp. 1-25
Author(s):  
Bohong Zhu ◽  
Youmin Chen ◽  
Qing Wang ◽  
Youyou Lu ◽  
Jiwu Shu
Keyword(s):  
High Speed ◽  
High Performance ◽  
File System ◽  
Direct Memory Access ◽  
File Systems ◽  
Distributed File Systems ◽  
Persistent Memory ◽  
Memory Modules ◽  
Non Volatile Memory ◽  
Volatile Memory

Non-volatile memory and remote direct memory access (RDMA) provide extremely high performance in storage and network hardware. However, existing distributed file systems strictly isolate file system and network layers, and the heavy layered software designs leave high-speed hardware under-exploited. In this article, we propose an RDMA-enabled distributed persistent memory file system, Octopus + , to redesign file system internal mechanisms by closely coupling non-volatile memory and RDMA features. For data operations, Octopus + directly accesses a shared persistent memory pool to reduce memory copying overhead, and actively fetches and pushes data all in clients to rebalance the load between the server and network. For metadata operations, Octopus + introduces self-identified remote procedure calls for immediate notification between file systems and networking, and an efficient distributed transaction mechanism for consistency. Octopus + is enabled with replication feature to provide better availability. Evaluations on Intel Optane DC Persistent Memory Modules show that Octopus + achieves nearly the raw bandwidth for large I/Os and orders of magnitude better performance than existing distributed file systems.

High Performance Storage for Big Data Analytics and Visualization

2018 ◽  
pp. 254-275
Author(s):  
Armando Fandango ◽  
William Rivera
Keyword(s):  
Big Data ◽  
High Speed ◽  
High Performance ◽  
File System ◽  
Predictive Analytics ◽  
Big Data Analytics ◽  
File Systems ◽  
Distributed Applications ◽  
System Level ◽  
File Formats

Scientific Big Data being gathered at exascale needs to be stored, retrieved and manipulated. The storage stack for scientific Big Data includes a file system at the system level for physical organization of the data, and a file format and input/output (I/O) system at the application level for logical organization of the data; both of them of high-performance variety for exascale. The high-performance file system is designed with concurrent access, high-speed transmission and fault tolerance characteristics. High-performance file formats and I/O are designed to allow parallel and distributed applications with easy and fast access to Big Data. These specialized file formats make it easier to store and access Big Data for scientific visualization and predictive analytics. This chapter provides a brief review of the characteristics of high-performance file systems such as Lustre and GPFS, and high-performance file formats such as HDF5, NetCDF, MPI-IO, and HDFS.

A high-performance distributed file system for large-scale concurrent HD video streams

2015 ◽  
Vol 27 (13) ◽  
pp. 3510-3522 ◽  
Author(s):  
Hancong Duan ◽  
Wenhan Zhan ◽  
Geyong Min ◽  
Hui Guo ◽  
Shengmei Luo
Keyword(s):  
High Performance ◽  
Large Scale ◽  
File System ◽  
Distributed File System ◽  
Video Streams

Performance Analysis of Lustre File System using High Performance Storage Devices

2016 ◽  
Vol 22 (4) ◽  
pp. 163-169 ◽  
Author(s):  
Jaehwan Lee ◽  
Donghun Koo ◽  
Kyungmin Park ◽  
Jiksoo Kim ◽  
Soonwook Hwang
Keyword(s):  
Performance Analysis ◽  
High Performance ◽  
File System ◽  
Storage Devices
Sign in / Sign up

Export Citation Format

Share Document