Offline and Online Algorithms for SSD Management

Flash-based solid state drives (SSDs) have gained a central role in the infrastructure of large-scale datacenters, as well as in commodity servers and personal devices. The main limitation of flash media is its inability to support update-in-place: after data has been written to a physical location, it has to be erased before new data can be written to it. Moreover, SSDs support read and write operations in granularity of pages, while erasures are performed on entire blocks, which often contain hundreds of pages. When erasing a block, any valid data it stores must be rewritten to a clean location. As an SSD eventually wears out with progressing number of erasures, the efficiency of the management algorithm has a significant impact on its endurance. In this paper we first formally define the SSD management problem. We then explore this problem from an algorithmic perspective, considering it in both offline and online settings. In the offline setting, we present a near-optimal algorithm that, given any input, performs a negligible number of rewrites (relative to the input length). We also discuss the hardness of the offline problem. In the online setting, we first consider algorithms that have no prior knowledge about the input. We prove that no deterministic algorithm outperforms the greedy algorithm in this setting, and discuss the possible benefit of randomization. We then augment our model, assuming that each request for a page arrives with a prediction of the next time the page is updated. We design an online algorithm that uses such predictions, and show that its performance improves as the prediction error decreases. We also show that the performance of our algorithm is never worse than that guaranteed by the greedy algorithm, even when the prediction error is large. We complement our theoretical findings with an empirical evaluation of our algorithms, comparing them with the state-of-the-art scheme. The results confirm that our algorithms exhibit an improved performance for a wide range of input traces.

Reused Shadow Recovery Scheme for Flash-based Edge Gateway Servers

Edge computing refers to decentralized computing technology to reduce cloud computing's overload or security problems that redirect local data to a central data center. Edge computing is emerging as a technology that complements cloud computing in an IoT environment where huge amounts of data are generated in real time. Recently, solid state drives using flash memory have recently been recognized as a suitable storage for massive IoT data services. In this study, we propose a new data recovery scheme based on shadow paging using flash memory for effective and safe data services in IoT edge gateways. The proposed scheme recycles invalidated old data blocks that are discarded when new data is stored. Thus, The proposed scheme minimizes the burden of additional storage space required to traditional shadow paging schemes, and reduces I/O performance degradation. Simulation results show that the space gain of the proposed scheme reaches even to 29%.

Leveraging NVMe SSDs for Building a Fast, Cost-effective, LSM-tree-based KV Store

Key-value (KV) stores support many crucial applications and services. They perform fast in-memory processing but are still often limited by I/O performance. The recent emergence of high-speed commodity non-volatile memory express solid-state drives (NVMe SSDs) has propelled new KV system designs that take advantage of their ultra-low latency and high bandwidth. Meanwhile, to switch to entirely new data layouts and scale up entire databases to high-end SSDs requires considerable investment. As a compromise, we propose SpanDB, an LSM-tree-based KV store that adapts the popular RocksDB system to utilize selective deployment of high-speed SSDs . SpanDB allows users to host the bulk of their data on cheaper and larger SSDs (and even hard disc drives with certain workloads), while relocating write-ahead logs (WAL) and the top levels of the LSM-tree to a much smaller and faster NVMe SSD. To better utilize this fast disk, SpanDB provides high-speed, parallel WAL writes via SPDK, and enables asynchronous request processing to mitigate inter-thread synchronization overhead and work efficiently with polling-based I/O. To ease the live data migration between fast and slow disks, we introduce TopFS, a stripped-down file system providing familiar file interface wrappers on top of SPDK I/O. Our evaluation shows that SpanDB simultaneously improves RocksDB's throughput by up to 8.8 \times and reduces its latency by 9.5–58.3%. Compared with KVell, a system designed for high-end SSDs, SpanDB achieves 96–140% of its throughput, with a 2.3–21.6 \times lower latency, at a cheaper storage configuration.

RocksDB: Evolution of Development Priorities in a Key-value Store Serving Large-scale Applications

This article is an eight-year retrospective on development priorities for RocksDB, a key-value store developed at Facebook that targets large-scale distributed systems and that is optimized for Solid State Drives (SSDs). We describe how the priorities evolved over time as a result of hardware trends and extensive experiences running RocksDB at scale in production at a number of organizations: from optimizing write amplification, to space amplification, to CPU utilization. We describe lessons from running large-scale applications, including that resource allocation needs to be managed across different RocksDB instances, that data formats need to remain backward- and forward-compatible to allow incremental software rollouts, and that appropriate support for database replication and backups are needed. Lessons from failure handling taught us that data corruption errors needed to be detected earlier and that data integrity protection mechanisms are needed at every layer of the system. We describe improvements to the key-value interface. We describe a number of efforts that in retrospect proved to be misguided. Finally, we describe a number of open problems that could benefit from future research.

Data Pattern Aware Reliability Enhancement Scheme for 3D Solid-State Drives

3D charge-trap (CT) NAND flash-based SSD has been used widely for its large capacity, low cost per bit, and high endurance. One-shot program (OSP) scheme, as a variation of incremental step pulse programming (ISPP) scheme, has been employed to program data for CT flash, whose program unit is the Word-Line (WL) instead of the page. The existing program optimization schemes either make trade-offs among program latency and reliability by adjusting the program step voltage on demand; or remap the most error-prone cell states to others by re-encoding programmed data. However, the data pattern, which represents the ratio of 1s in data values, has not been thoroughly studied. In this paper, we observe that most small files do not contain uniform 1s and 0s among these common file types (i.e., image, audio, text, executable file), leading to programming WL cells in different states unevenly. Some cell states dominate over the WL, while others are not. Based on this observation, we propose a flexible reliability enhancement scheme based on the OSP scheme. This scheme programs the cells into different states with varied , i.e., these cells in one state, whose number is the largest in one WL, are programmed with a fine-grained (namely slow write). In contrast, the minority are programmed with a coarse-grained (namely fast write). So the reliability is improved due to averaging the major enhanced cells with the minor degraded cells without program latency overhead. A series of experiments have been conducted, and the results indicate that the proposed scheme achieves 34% read performance improvement and 16% lifetime elongation on average.

A Cache Policy Based on Request Association Analysis for Reliable NAND-Based Storage Systems

Compared with the traditional hard-disk drives (HDDs), solid-state drives (SSDs) have adopted NAND flash memory and become the current popular storage devices. However, when the free space in NAND flash memory is not enough, the garbage collection will be triggered to recycle the free space. The activities of the garbage collection include a large amount of data written and time-consuming erase operations that can reduce the performance of NAND flash memory. Therefore, DRAM is usually added to NAND flash memory as cache to store frequently used data. The typical cache methods mainly utilize the data characteristics of temporal locality and spatial locality to keep the frequently used data in the cache as much as possible. In addition, we find that there are not only temporal/spatial locality, but also certain associations between the accessed data. Therefore, we suggest that a cache policy should not only consider the temporal/spatial locality but also consider the association relationship between the accessed data to improve the cache hit ratio. In the paper, we will propose a cache policy based on request association analysis for reliable NAND-based storage systems. According to the experimental results, the cache hit ratio of the proposed method can be increased significantly when compared with the typical cache methods.