Dynamic Index Management in Log-Structured Merge Storage

Abstract

Many modern key–value stores adopt Log-Structured Merge-trees (LSMs) to support high write throughput. A critical component of an LSM is the memtable, and its internal index structure (memtable type) affects system performance as the memtable serves all writes and a substantial fraction of reads. Existing systems are generally designed with a static memtable index structure. However, such a static design cannot adjust to workload shifts as it is unable to consider workload-dependent characteristics of different memtable types, leading to performance degradation. To address this shortcoming, we propose Dynamic Memtable Index Organization (DIO), a cost-driven framework that dynamically adapts the memtable type according to workload characteristics. DIO models query costs by dissecting queries into primitive memory operations, predicts the workload cost under multiple memtable types, and selects a memtable type that minimizes cost. DIO performs memtable (type) transitions at meaningful memtable lifecycle boundaries, enabling efficient adaptation with minimal disruption to system execution. DIO incorporates mechanisms for early workload-shift detection and on-the-fly memtable conversion, allowing the system to rapidly converge to the most suitable memtable type after workload changes. We implement DIO in RocksDB and show that, across diverse dynamic workloads, it improves post-transition throughput by up to several orders of magnitude over unmodified RocksDB with negligible runtime overhead.