Comparison and evaluation of state-of-the-art LSM merge policies
AbstractModern NoSQL database systems use log-structured merge (LSM) storage architectures to support high write throughput. LSM architectures aggregate writes in a mutableMemTable(stored in memory), which is regularly flushed to disk, creating a new immutable file called anSSTable. Some of the SSTables are chosen to be periodicallymerged—replaced with a single SSTable containing their union. Amergepolicy(a.k.a. compaction policy) specifies when to do merges and which SSTables to combine. Abounded depthmerge policy is one that guarantees that the number of SSTables never exceeds a given parameterk, typically in the range 3–10. Bounded depth policies are useful in applications where low read latency is crucial, but they and their underlying combinatorics are not yet well understood. This paper compares several bounded depth policies, including representative policies from industrial NoSQL databases and two new ones based on recent theoretical modeling, as well as the standard Tiered policy and Leveled policy. The results validate the proposed theoretical model and show that, compared to the existing policies, the newly proposed policies can have substantially lower write amplification with comparable read amplification.
