scholarly journals Composition and Realization of Source-to-Sink High-Performance Flows: File Systems, Storage, Hosts, LAN and WAN

2016 ◽  
Author(s):  
Chase Qishi Wu
2021 ◽  
Vol 17 (3) ◽  
pp. 1-25
Author(s):  
Bohong Zhu ◽  
Youmin Chen ◽  
Qing Wang ◽  
Youyou Lu ◽  
Jiwu Shu

Non-volatile memory and remote direct memory access (RDMA) provide extremely high performance in storage and network hardware. However, existing distributed file systems strictly isolate file system and network layers, and the heavy layered software designs leave high-speed hardware under-exploited. In this article, we propose an RDMA-enabled distributed persistent memory file system, Octopus + , to redesign file system internal mechanisms by closely coupling non-volatile memory and RDMA features. For data operations, Octopus + directly accesses a shared persistent memory pool to reduce memory copying overhead, and actively fetches and pushes data all in clients to rebalance the load between the server and network. For metadata operations, Octopus + introduces self-identified remote procedure calls for immediate notification between file systems and networking, and an efficient distributed transaction mechanism for consistency. Octopus + is enabled with replication feature to provide better availability. Evaluations on Intel Optane DC Persistent Memory Modules show that Octopus + achieves nearly the raw bandwidth for large I/Os and orders of magnitude better performance than existing distributed file systems.


2020 ◽  
Vol 26 (1) ◽  
pp. 89-106
Author(s):  
Kohei Hiraga ◽  
Osamu Tatebe ◽  
Hideyuki Kawashima

Metadata performance scalability is critically important in high-performance computing when accessing many small files from millions of clients. This paper proposes a design of a scalable distributed metadata server, PPMDS, for parallel file systems using multiple key-value servers. In PPMDS, hierarchical namespace of a file system is efficiently managed by multiple servers. Multiple entries can be atomically updated using a nonblocking distributed transaction based on an algorithm of dynamic software transactional memory. This paper also proposes optimizations to further improve the metadata performance by introducing a server-side transaction processing, multiple readers, and a shared lock mode, which reduce the number of remote procedure calls and prevent unnecessary blocking. Performance evaluation shows the scalable performance up to 3 servers, and achieves 62,000 operations per second, which is 2.58x performance improvement compared to a single metadata performance.


Author(s):  
Armando Fandango ◽  
William Rivera

Scientific Big Data being gathered at exascale needs to be stored, retrieved and manipulated. The storage stack for scientific Big Data includes a file system at the system level for physical organization of the data, and a file format and input/output (I/O) system at the application level for logical organization of the data; both of them of high-performance variety for exascale. The high-performance file system is designed with concurrent access, high-speed transmission and fault tolerance characteristics. High-performance file formats and I/O are designed to allow parallel and distributed applications with easy and fast access to Big Data. These specialized file formats make it easier to store and access Big Data for scientific visualization and predictive analytics. This chapter provides a brief review of the characteristics of high-performance file systems such as Lustre and GPFS, and high-performance file formats such as HDF5, NetCDF, MPI-IO, and HDFS.


2020 ◽  
Vol 35 (1) ◽  
pp. 4-26 ◽  
Author(s):  
André Brinkmann ◽  
Kathryn Mohror ◽  
Weikuan Yu ◽  
Philip Carns ◽  
Toni Cortes ◽  
...  

VLSI Design ◽  
1998 ◽  
Vol 7 (1) ◽  
pp. 15-30
Author(s):  
Gustavo E. Téllez ◽  
Majid Sarrafzadeh

Given a set of terminals on the plane N={s,ν1,…,νn}, with a source terminal s, a Rectilinear Distance-Preserving Tree (RDPT) T(V, E) is defined as a tree rooted at s, connecting all terminals in N. An RDPT has the property that the length of every source to sink path is equal to the rectilinear distance between that source and sink. A Min- Cost Rectilinear Distance-Preserving Tree (MRDPT) minimizes the total wire length while maintaining minimal source to sink linear delay, making it suitable for high performance interconnect applications.This paper studies problems in the construction of RDPTs, including the following contributions. A new exact algorithm for a restricted version of the problem in one quadrant with O(n2) time complexity is proposed. A novel heuristic algorithm, which uses optimally solvable sub-problems, is proposed for the problem in a single quadrant. The average and worst-case time complexity for the proposed heuristic algorithm are O(n3/2) and O(n3), respectively. A 2-approximation of the quadrant merging problem is proposed. The proposed algorithm has time complexity O(α2T(n)+α3) for any constant α > 1, where T(n) is the time complexity of the solution of the RDPT problem on one quadrant. This result improves over the best previous quadrant merging solution which has O(n2T(n)+n3) time complexity.We test our algorithms on randomly uniform point sets and compare our heuristic RDPT construction against a Minimum Cost Rectilinear Steiner (MRST) tree approximation algorithm. Our results show that RDPTs are competitive with Steiner trees in total wire-length when the number of terminals is less than 32. This result makes RDPTs suitable for VLSI routing applications. We also compare our algorithm to the Rao-Shor RDPT approximation algorithm obtaining improvements of up to 10% in total wirelength. These comparisons show that the algorithms proposed herein produce promising results.


2016 ◽  
Vol 43 (5) ◽  
pp. 524-530
Author(s):  
Jiwon Kim ◽  
Jungsik Choi ◽  
Hwansoo Han

Sign in / Sign up

Export Citation Format

Share Document