RocksDB: Evolution of Development Priorities in a Key-value Store Serving Large-scale Applications

This article is an eight-year retrospective on development priorities for RocksDB, a key-value store developed at Facebook that targets large-scale distributed systems and that is optimized for Solid State Drives (SSDs). We describe how the priorities evolved over time as a result of hardware trends and extensive experiences running RocksDB at scale in production at a number of organizations: from optimizing write amplification, to space amplification, to CPU utilization. We describe lessons from running large-scale applications, including that resource allocation needs to be managed across different RocksDB instances, that data formats need to remain backward- and forward-compatible to allow incremental software rollouts, and that appropriate support for database replication and backups are needed. Lessons from failure handling taught us that data corruption errors needed to be detected earlier and that data integrity protection mechanisms are needed at every layer of the system. We describe improvements to the key-value interface. We describe a number of efforts that in retrospect proved to be misguided. Finally, we describe a number of open problems that could benefit from future research.

Integrity protection method for trusted data of IOT nodes based on transfer learning

In order to overcome the problems of high data storage occupancy and long encryption time in traditional integrity protection methods for trusted data of IOT node, this paper proposes an integrity protection method for trusted data of IOT node based on transfer learning. Through the transfer learning algorithm, the data characteristics of the IOT node is obtained, the feature mapping function in the common characteristics of the node data is set to complete the classification of the complete data and incomplete data in the IOT nodes. The data of the IOT nodes is input into the data processing database to verify its security, eliminate the node data with low security, and integrate the security data and the complete data. On this basis, homomorphic encryption algorithm is used to encrypt the trusted data of IOT nodes, and embedded processor is added to the IOT to realize data integrity protection. The experimental results show that: after using the proposed method to protect the integrity of trusted data of IOT nodes, the data storage occupancy rate is only about 3.5%, the shortest time-consuming of trusted data encryption of IOT nodes is about 3 s, and the work efficiency is high.