scholarly journals Grouping-Based Consistency Protocol Design for End-Edge-Cloud Hierarchical Storage System

IEEE Access ◽  
2020 ◽  
Vol 8 ◽  
pp. 8959-8973 ◽  
Author(s):  
Shushi Gu ◽  
Yizhen Wang ◽  
Ye Wang ◽  
Qinyu Zhang ◽  
Xue Qin
Keyword(s):  
Storage System ◽  
Protocol Design ◽  
Hierarchical Storage ◽  
Consistency Protocol

Statistical Placement on Hierarchical Storage Systems

2011 ◽  
pp. 161-166
Author(s):  
Phillip K.C. Tse
Keyword(s):  
Storage Systems ◽  
Storage System ◽  
Access Time ◽  
Placement Method ◽  
Statistical Strategy ◽  
Hierarchical Storage ◽  
The Mean ◽  
Statistical Time ◽  
Strategy Changes

We have described the contiguous placement in the previous chapter and the statistical strategy to place objects on disks in Chapter IV. In this chapter, we describe the statistical strategy to place them on hierarchical storage systems. The objective of the data placement methods is to minimize the time to access object from the hierarchical storage system. The statistical strategy changes the statistical time to access objects so that the mean access time is optimal. The objective of the frequency based placement method is to differentiate objects according to their access frequencies. The objects that are more frequently accessed are placed in the more convenient locations. The objects that are less frequently accessed are placed in the less convenient locations. We will describe the frequency based placement method in the next section. Afterwards, we will analyze its performance. Last, we summarize this chapter.

The HP AutoRAID hierarchical storage system

1996 ◽  
Vol 14 (1) ◽  
pp. 108-136 ◽  
Author(s):  
John Wilkes ◽  
Richard Golding ◽  
Carl Staelin ◽  
Tim Sullivan
Keyword(s):  
Storage System ◽  
Hierarchical Storage

A Large Site Centralized Data Classification Strategy Based on User Value

2014 ◽  
Vol 1030-1032 ◽  
pp. 1619-1622
Author(s):  
Bing Xin Zhu ◽  
Jing Tao Li
Keyword(s):  
High Performance ◽  
Large Scale ◽  
Storage System ◽  
Data Classification ◽  
Data Access ◽  
Heat Index ◽  
Storage Devices ◽  
Hierarchical Storage ◽  
Classification Strategy ◽  
User Value

In large-scale storage system, variety of calculations, transfer, and storage devices both in performance and in characteristics such as reliability, there are physical differences. While operational load data access for storage devices is also not uniform, there is a big difference in space and time. If all the data is stored in the high-performance equipment is unrealistic and unwise. Hierarchical storage concept effectively solves this problem. It is able to monitor the data access loads, and depending on the load and application requirements based on storage resources optimally configure properties [1]. Traditional classification policy is generally against file data, based on frequency of access to files, file IO heat index for classification. This paper embarks from the website user value concept, aiming at the disadvantages of traditional data classification strategy, puts forward the centralized data classification strategy based on user value.

Transfer Data from NetCDF on Hierarchical Storage to Zarr on Object Storage: CMIP6 Climate Data Use Case

2021 ◽  
Author(s):  
Marco Kulüke ◽  
Fabian Wachsmann ◽  
Georg Leander Siemund ◽  
Hannes Thiemann ◽  
Stephan Kindermann
Keyword(s):  
Storage Systems ◽  
Storage System ◽  
File Systems ◽  
Data Access ◽  
Data Retrieval ◽  
Use Case ◽  
Climate Data ◽  
Object Storage ◽  
Hierarchical Storage ◽  
Data Transfers

<p>This study provides a guidance to data providers on how to transfer existing NetCDF data from a hierarchical storage system into Zarr to an object storage system.</p><p>In recent years, object storage systems became an alternative to traditional hierarchical file systems, because they are easily scalable and offer faster data retrieval, as compared to hierarchical storage systems.</p><p>Earth system sciences, and climate science in particular, handle large amounts of data. These data usually are represented as multi-dimensional arrays and traditionally stored in netCDF format on hierarchical file systems. However, the current netCDF-4 format is not yet optimized for object storage systems. NetCDF data transfers from an object storage can only be conducted on file level which results in heavy download volumes. An improvement to mitigate this problem can be the Zarr format, which reduces data transfers, due to the direct chunk and meta data access and hence increases the input/output operation speed in parallel computing environments.</p><p>As one of the largest climate data providers worldwide, the German Climate Computing Center (DKRZ) continuously works towards efficient ways to make data accessible for the user. This use case shows the conversion and the transfer of a subset of the Coupled Model Intercomparison Project Phase 6 (CMIP6) climate data archive from netCDF on the hierarchical file system into Zarr to the OpenStack object store, known as Swift, by using the Zarr Python package. Conclusively, this study will evaluate to what extent Zarr formatted climate data on an object storage system is a meaningful addition to the existing high performance computing environment of the DKRZ.</p>

The HP AutoRAID hierarchical storage system

1995 ◽  
Vol 29 (5) ◽  
pp. 96-108 ◽  
Author(s):  
J. Wilkes ◽  
R. Golding ◽  
C. Staelin ◽  
T. Sullivan
Keyword(s):  
Storage System ◽  
Hierarchical Storage
Sign in / Sign up

Export Citation Format

Share Document