scholarly journals Performance Evaluation of NVMe-over-TCP Using Journaling File Systems in International WAN

Electronics ◽  
2021 ◽  
Vol 10 (20) ◽  
pp. 2486
Author(s):  
Se-young Yu
Keyword(s):  
File System ◽  
Data Transfer ◽  
Storage System ◽  
File Systems ◽  
Data Access ◽  
Streaming Data ◽  
Fine Tuning ◽  
Long Distance ◽  
Data Movement ◽  
Remote Data

Distributing Big Data for science is pushing the capabilities of networks and computing systems. However, the fundamental concept of copying data from one machine to another has not been challenged in collaborative science. As recent storage system development uses modern fabrics to provide faster remote data access with lower overhead, traditional data movement using Data Transfer Nodes must cope with the paradigm shift from a store-and-forward model to streaming data with direct storage access over the networks. This study evaluates NVMe-over-TCP (NVMe-TCP) in a long-distance network using different file systems and configurations to characterize remote NVMe file system access performance in MAN and WAN data moving scenarios. We found that NVMe-TCP is more suitable for remote data read than remote data write over the networks, and using RAID0 can significantly improve performance in a long-distance network. Additionally, a fine-tuning file system can improve remote write performance in DTNs with a long-distance network.

scholarly journals Design and Evaluation of a Simple Data Interface for Efficient Data Transfer across Diverse Storage

2021 ◽  
Vol 6 (1) ◽  
pp. 1-25
Author(s):  
Zhengchun Liu ◽  
Rajkumar Kettimuthu ◽  
Joaquin Chung ◽  
Rachana Ananthakrishnan ◽  
Michael Link ◽  
...  
Keyword(s):  
Performance Optimization ◽  
Data Exchange ◽  
Data Transfer ◽  
Storage Systems ◽  
Distributed Storage ◽  
Storage System ◽  
File Systems ◽  
Modern Science ◽  
Data Access ◽  
Third Party

Modern science and engineering computing environments often feature storage systems of different types, from parallel file systems in high-performance computing centers to object stores operated by cloud providers. To enable easy, reliable, secure, and performant data exchange among these different systems, we propose Connector, a plug-able data access architecture for diverse, distributed storage. By abstracting low-level storage system details, this abstraction permits a managed data transfer service (Globus, in our case) to interact with a large and easily extended set of storage systems. Equally important, it supports third-party transfers: that is, direct data transfers from source to destination that are initiated by a third-party client but do not engage that third party in the data path. The abstraction also enables management of transfers for performance optimization, error handling, and end-to-end integrity. We present the Connector design, describe implementations for different storage services, evaluate tradeoffs inherent in managed vs. direct transfers, motivate recommended deployment options, and propose a model-based method that allows for easy characterization of performance in different contexts without exhaustive benchmarking.

Performance Measurement Technique of Cloud Storage System

2013 ◽  
Vol 760-762 ◽  
pp. 1197-1201
Author(s):  
Qin Lu He ◽  
Zhan Huai Li ◽  
Le Xiao Wang ◽  
Hui Feng Wang ◽  
Jian Sun
Keyword(s):  
Cloud Storage ◽  
File System ◽  
Storage System ◽  
File Systems ◽  
Test Methods ◽  
Theory Analysis ◽  
Network File System ◽  
Cloud Storage Environment ◽  
System File ◽  
Reasonable Test

Researches on technologies about testing aggregate bandwidth of file systems in cloud storage systems. Through the memory file system, network file system, parallel file system theory analysis, according to the cloud storage system polymerization bandwidth and concept, developed to cloud storage environment file system polymerization bandwidth test software called FSPoly. In this paper, use FSpoly to luster file system testing, find reasonable test methods, and then evaluations latest development in cloud storage system file system performance by using FSPoly.

scholarly journals An Approach for Evaluating and Mitigating Intra-Application I/O Performance Variability Over Parallel File Systems

2019 ◽  
Author(s):  
Eduardo Inacio ◽  
Mario Antonio Dantas
Keyword(s):  
Load Distribution ◽  
Large Scale ◽  
File System ◽  
Storage System ◽  
Research Work ◽  
File Systems ◽  
Distributed Applications ◽  
Evaluation Tool ◽  
Performance Variability ◽  
Parallel File

To meet ever increasing capacity and performance requirements of emerging data-intensive applications, highly distributed and multilayered back-end storage systems have been employed in large-scale high performance computing (HPC) environments. A main component of these storage infrastructures is the parallel file system (PFS), a especially designed file system for absorbing bulk data transfers from applications with thousands of concurrent processes. Load distribution on PFS data servers compose a major source of intra-application input/output (I/O) performance variability. Albeit mitigating variability is desirable, as it is known to harm application-perceived performance, understanding and dealing with I/O performance variability in such complex environments remains a challenging task. In this research, a differentiated approach for evaluating and mitigating intra-application I/O performance variability over PFSs is proposed. More specifically, from the evaluation perspective, a comprehensive approach combining complementary methods is proposed. An analytical model proposal, named DTSMaxLoad, provides estimates for the maximum load in a PFS data server. To complement DTSMaxLoad, modeling conditions and mechanisms hard to represent analytically, the Parallel I/O and Storage System (PIOSS) simulation model was proposed. Finally, for experimental evaluation over real environments, a flexible and distributed I/O performance evaluation tool, coined as IOR-Extended (IORE), was proposed. Furthermore, a high-level file distribution approach for PFSs, called N-N Round-Robin (N2R2), was proposed focusing on mitigating I/O performance variability for distributed applications where each process accesses an individual and independent file. An extensive experimental effort, including measurements on real environments, was conducted in this research work for evaluating each of the proposed approaches. In summary, this evaluation indicated both DTSMaxLoad and PIOSS modeling proposals can represent load distribution behavior on PFSs with significant fidelity. Moreover, results demonstrated N2R2 successfully reduced intra-application I/O performance variability for 270 distinct experimental scenarios, which, ultimately, translated into overall application I/O performance Improvements.

Transfer Data from NetCDF on Hierarchical Storage to Zarr on Object Storage: CMIP6 Climate Data Use Case

2021 ◽  
Author(s):  
Marco Kulüke ◽  
Fabian Wachsmann ◽  
Georg Leander Siemund ◽  
Hannes Thiemann ◽  
Stephan Kindermann
Keyword(s):  
Storage Systems ◽  
Storage System ◽  
File Systems ◽  
Data Access ◽  
Data Retrieval ◽  
Use Case ◽  
Climate Data ◽  
Object Storage ◽  
Hierarchical Storage ◽  
Data Transfers

<p>This study provides a guidance to data providers on how to transfer existing NetCDF data from a hierarchical storage system into Zarr to an object storage system.</p><p>In recent years, object storage systems became an alternative to traditional hierarchical file systems, because they are easily scalable and offer faster data retrieval, as compared to hierarchical storage systems.</p><p>Earth system sciences, and climate science in particular, handle large amounts of data. These data usually are represented as multi-dimensional arrays and traditionally stored in netCDF format on hierarchical file systems. However, the current netCDF-4 format is not yet optimized for object storage systems. NetCDF data transfers from an object storage can only be conducted on file level which results in heavy download volumes. An improvement to mitigate this problem can be the Zarr format, which reduces data transfers, due to the direct chunk and meta data access and hence increases the input/output operation speed in parallel computing environments.</p><p>As one of the largest climate data providers worldwide, the German Climate Computing Center (DKRZ) continuously works towards efficient ways to make data accessible for the user. This use case shows the conversion and the transfer of a subset of the Coupled Model Intercomparison Project Phase 6 (CMIP6) climate data archive from netCDF on the hierarchical file system into Zarr to the OpenStack object store, known as Swift, by using the Zarr Python package. Conclusively, this study will evaluate to what extent Zarr formatted climate data on an object storage system is a meaningful addition to the existing high performance computing environment of the DKRZ.</p>

Exploring Efficient Architectures on Remote In-Memory NVM over RDMA

2021 ◽  
Vol 20 (5s) ◽  
pp. 1-20
Author(s):  
Qingfeng Zhuge ◽  
Hao Zhang ◽  
Edwin Hsing-Mean Sha ◽  
Rui Xu ◽  
Jun Liu ◽  
...  
Keyword(s):  
High Performance ◽  
File System ◽  
File Systems ◽  
Data Access ◽  
Main Memory ◽  
Memory Modules ◽  
Significant Performance ◽  
Architectural Structures ◽  
Memory Accesses ◽  
Careful Design

Efficiently accessing remote file data remains a challenging problem for data processing systems. Development of technologies in non-volatile dual in-line memory modules (NVDIMMs), in-memory file systems, and RDMA networks provide new opportunities towards solving the problem of remote data access. A general understanding about NVDIMMs, such as Intel Optane DC Persistent Memory (DCPM), is that they expand main memory capacity with a cost of multiple times lower performance than DRAM. With an in-depth exploration presented in this paper, however, we show an interesting finding that the potential of NVDIMMs for high-performance, remote in-memory accesses can be revealed through careful design. We explore multiple architectural structures for accessing remote NVDIMMs in a real system using Optane DCPM, and compare the performance of various structures. Experiments are conducted to show significant performance gaps among different ways of using NVDIMMs as memory address space accessible through RDMA interface. Furthermore, we design and implement a prototype of user-level, in-memory file system, RIMFS, in the device DAX mode on Optane DCPM. By comparing against the DAX-supported Linux file system, Ext4-DAX, we show that the performance of remote reads on RIMFS over RDMA is 11.44 higher than that on a remote Ext4-DAX on average. The experimental results also show that the performance of remote accesses on RIMFS is maintained on a heavily loaded data server with CPU utilization as high as 90%, while the performance of remote reads on Ext4-DAX is significantly reduced by 49.3%, and the performance of local reads on Ext4-DAX is even more significantly reduced by 90.1%. The performance comparisons of writes exhibit the same trends.

XtreemFS

2013 ◽  
pp. 267-285 ◽  
Author(s):  
Jan Stender ◽  
Michael Berlin ◽  
Alexander Reinefeld
Keyword(s):  
Data Storage ◽  
Large Scale ◽  
File System ◽  
Fault Tolerant ◽  
File Systems ◽  
Data Access ◽  
Comprehensive Overview ◽  
Cloud Providers ◽  
The Face ◽  
Cloud Users

Cloud computing poses new challenges to data storage. While cloud providers use shared distributed hardware, which is inherently unreliable and insecure, cloud users expect their data to be safely and securely stored, available at any time, and accessible in the same way as their locally stored data. In this chapter, the authors present XtreemFS, a file system for the cloud. XtreemFS reconciles the need of cloud providers for cheap scale-out storage solutions with that of cloud users for a reliable, secure, and easy data access. The main contributions of the chapter are: a description of the internal architecture of XtreemFS, which presents an approach to build large-scale distributed POSIX-compliant file systems on top of cheap, off-the-shelf hardware; a description of the XtreemFS security infrastructure, which guarantees an isolation of individual users despite shared and insecure storage and network resources; a comprehensive overview of replication mechanisms in XtreemFS, which guarantee consistency, availability, and durability of data in the face of component failures; an overview of the snapshot infrastructure of XtreemFS, which allows to capture and freeze momentary states of the file system in a scalable and fault-tolerant fashion. The authors also compare XtreemFS with existing solutions and argue for its practicability and potential in the cloud storage market.

Resource Allocation for Multi Access MIMO Systems

2011 ◽  
Vol 3 (3) ◽  
pp. 36-50
Author(s):  
Shailendra Mishra ◽  
D. S. Chauhan
Keyword(s):  
Resource Allocation ◽  
High Speed ◽  
New Technologies ◽  
Data Transfer ◽  
Mimo Systems ◽  
Data Access ◽  
Streaming Data ◽  
The Other ◽  
Packet Data ◽  
Multi Access

In this paper, the authors discuss the emergence of new technologies related to the topic of the high-speed packet data access in wireless networks. The authors propose an algorithm for MIMO systems that optimizes the number of the transmit antennas according to the user’s QoS. Scheduling performance under two types of traffic modes is also discussed: one is voice or web-browsing and the other is for data transfer and streaming data.

scholarly journals ExSeisDat: A set of parallel I/O and workflow libraries for petroleum seismology

2018 ◽  
Vol 73 ◽  
pp. 74 ◽  
Author(s):  
Meghan A. Fisher ◽  
Pádraig Ó. Conbhuí ◽  
Cathal Ó. Brion ◽  
Jean-Thomas Acquaviva ◽  
Seán Delaney ◽  
...  
Keyword(s):  
Seismic Data ◽  
Memory Management ◽  
File System ◽  
File Systems ◽  
Data Access ◽  
System Level ◽  
Data Set ◽  
Extreme Scale ◽  
Data Access Patterns ◽  
Access Patterns

Seismic data-sets are extremely large and are broken into data files, ranging in size from 100s of GiBs to 10s of TiBs and larger. The parallel I/O for these files is complex due to the amount of data along with varied and multiple access patterns within individual files. Properties of legacy file formats, such as the de-facto standard SEG-Y, also contribute to the decrease in developer productivity while working with these files. SEG-Y files embed their own internal layout which could lead to conflict with traditional, file-system-level layout optimization schemes. Additionally, as seismic files continue to increase in size, memory bottlenecks will be exacerbated, resulting in the need for smart I/O optimization not only to increase the efficiency of read/writes, but to manage memory usage as well. The ExSeisDat (Extreme-Scale Seismic Data) set of libraries addresses these problems through the development and implementation of easy to use, object oriented libraries that are portable and open source with bindings available in multiple languages. The lower level parallel I/O library, ExSeisPIOL (Extreme-Scale Seismic Parallel I/O Library), targets SEG-Y and other proprietary formats, simplifying I/O by internally interfacing MPI-I/O and other I/O interfaces. The I/O is explicitly handled; end users only need to define the memory limits, decomposition of I/O across processes, and data access patterns when reading and writing data. ExSeisPIOL bridges the layout gap between the SEG-Y file structure and file system organization. The higher level parallel seismic workflow library, ExSeisFlow (Extreme-Scale Seismic workFlow), leverages ExSeisPIOL, further simplifying I/O by implicitly handling all I/O parameters, thus allowing geophysicists to focus on domain-specific development. Operations in ExSeisFlow focus on prestack processing and can be performed on single traces, individual gathers, and across entire surveys, including out of core sorting, binning, filtering, and transforming. To optimize memory management, the workflow only reads in data pertinent to the operations being performed instead of an entire file. A smart caching system manages the read data, discarding it when no longer needed in the workflow. As the libraries are optimized to handle spatial and temporal locality, they are a natural fit to burst buffer technologies, particularly DDN’s Infinite Memory Engine (IME) system. With appropriate access semantics or through the direct exploitation of the low-level interfaces, the ExSeisDat stack on IME delivers a significant improvement to I/O performance over standalone parallel file systems like Lustre.

Using Hadoop Distributed and Deduplicated File System (HD2FS) in Astronomy

2019 ◽  
Vol 15 (S367) ◽  
pp. 464-466
Author(s):  
Paul Bartus
Keyword(s):  
Data Storage ◽  
Storage Capacity ◽  
File System ◽  
Storage Systems ◽  
Storage System ◽  
File Systems ◽  
Distributed File System ◽  
Distributed File Systems ◽  
Output Performance ◽  
Hadoop Distributed File System

AbstractDuring the last years, the amount of data has skyrocketed. As a consequence, the data has become more expensive to store than to generate. The storage needs for astronomical data are also following this trend. Storage systems in Astronomy contain redundant copies of data such as identical files or within sub-file regions. We propose the use of the Hadoop Distributed and Deduplicated File System (HD2FS) in Astronomy. HD2FS is a deduplication storage system that was created to improve data storage capacity and efficiency in distributed file systems without compromising Input/Output performance. HD2FS can be developed by modifying existing storage system environments such as the Hadoop Distributed File System. By taking advantage of deduplication technology, we can better manage the underlying redundancy of data in astronomy and reduce the space needed to store these files in the file systems, thus allowing for more capacity per volume.

Resource Allocation for Multi Access MIMO Systems

2012 ◽  
pp. 221-235
Author(s):  
Shailendra Mishra ◽  
Durg Singh Chauhan
Keyword(s):  
Resource Allocation ◽  
High Speed ◽  
New Technologies ◽  
Data Transfer ◽  
Mimo Systems ◽  
Data Access ◽  
Streaming Data ◽  
The Other ◽  
Packet Data ◽  
Multi Access

In this paper, the authors discuss the emergence of new technologies related to the topic of the high-speed packet data access in wireless networks. The authors propose an algorithm for MIMO systems that optimizes the number of the transmit antennas according to the user’s QoS. Scheduling performance under two types of traffic modes is also discussed: one is voice or web-browsing and the other is for data transfer and streaming data.

Sign in / Sign up

Export Citation Format

Share Document