Consider archived WALs for deletion more frequently (facebook#12069)

Summary:
Fixes facebook#11000.

That issue pointed out that RocksDB was slow to delete archived WALs in case time-based and size-based expiration were enabled, and the time-based threshold (`WAL_ttl_seconds`) was small. This PR prevents the delay by taking into account `WAL_ttl_seconds` when deciding the frequency to process archived WALs for deletion.

Pull Request resolved: facebook#12069

Reviewed By: pdillinger

Differential Revision: D51262589

Pulled By: ajkr

fbshipit-source-id: e65431a06ee96f4c599ba84a27d1aedebecbb003

db/wal_manager.cc

@@ -153,10 +153,11 @@ void WalManager::PurgeObsoleteWALFiles() {
     return;
   }
   uint64_t const now_seconds = static_cast<uint64_t>(current_time);
-  uint64_t const time_to_check = (ttl_enabled && !size_limit_enabled)
-                                     ? db_options_.WAL_ttl_seconds / 2
-                                     : kDefaultIntervalToDeleteObsoleteWAL;
-
+  uint64_t const time_to_check =
+      ttl_enabled
+          ? std::min(kDefaultIntervalToDeleteObsoleteWAL,
+                     std::max(uint64_t{1}, db_options_.WAL_ttl_seconds / 2))
+          : kDefaultIntervalToDeleteObsoleteWAL;
   if (purge_wal_files_last_run_ + time_to_check > now_seconds) {
     return;
   }

include/rocksdb/options.h

@@ -814,18 +814,23 @@ struct DBOptions {
   // Number of shards used for table cache.
   int table_cache_numshardbits = 6;
 
-  // The following two fields affect how archived logs will be deleted.
-  // 1. If both set to 0, logs will be deleted asap and will not get into
-  //    the archive.
-  // 2. If WAL_ttl_seconds is 0 and WAL_size_limit_MB is not 0,
-  //    WAL files will be checked every 10 min and if total size is greater
-  //    then WAL_size_limit_MB, they will be deleted starting with the
-  //    earliest until size_limit is met. All empty files will be deleted.
-  // 3. If WAL_ttl_seconds is not 0 and WAL_size_limit_MB is 0, then
-  //    WAL files will be checked every WAL_ttl_seconds / 2 and those that
-  //    are older than WAL_ttl_seconds will be deleted.
-  // 4. If both are not 0, WAL files will be checked every 10 min and both
-  //    checks will be performed with ttl being first.
+  // The following two fields affect when WALs will be archived and deleted.
+  //
+  // When both are zero, obsolete WALs will not be archived and will be deleted
+  // immediately. Otherwise, obsolete WALs will be archived prior to deletion.
+  //
+  // When `WAL_size_limit_MB` is nonzero, archived WALs starting with the
+  // earliest will be deleted until the total size of the archive falls below
+  // this limit. All empty WALs will be deleted.
+  //
+  // When `WAL_ttl_seconds` is nonzero, archived WALs older than
+  // `WAL_ttl_seconds` will be deleted.
+  //
+  // When only `WAL_ttl_seconds` is nonzero, the frequency at which archived
+  // WALs are deleted is every `WAL_ttl_seconds / 2` seconds. When only
+  // `WAL_size_limit_MB` is nonzero, the deletion frequency is every ten
+  // minutes. When both are nonzero, the deletion frequency is the minimum of
+  // those two values.
   uint64_t WAL_ttl_seconds = 0;
   uint64_t WAL_size_limit_MB = 0;
 

java/src/main/java/org/rocksdb/DBOptionsInterface.java

@@ -615,21 +615,24 @@ public interface DBOptionsInterface<T extends DBOptionsInterface<T>> {
   int tableCacheNumshardbits();
 
   /**
-   * {@link #walTtlSeconds()} and {@link #walSizeLimitMB()} affect how archived logs
-   * will be deleted.
-   * <ol>
-   * <li>If both set to 0, logs will be deleted asap and will not get into
-   * the archive.</li>
-   * <li>If WAL_ttl_seconds is 0 and WAL_size_limit_MB is not 0,
-   *    WAL files will be checked every 10 min and if total size is greater
-   *    then WAL_size_limit_MB, they will be deleted starting with the
-   *    earliest until size_limit is met. All empty files will be deleted.</li>
-   * <li>If WAL_ttl_seconds is not 0 and WAL_size_limit_MB is 0, then
-   *    WAL files will be checked every WAL_ttl_seconds / 2 and those that
-   *    are older than WAL_ttl_seconds will be deleted.</li>
-   * <li>If both are not 0, WAL files will be checked every 10 min and both
-   *    checks will be performed with ttl being first.</li>
-   * </ol>
+   * {@link #walTtlSeconds()} and {@link #walSizeLimitMB()} affect when WALs
+   * will be archived and deleted.
+   *
+   * When both are zero, obsolete WALs will not be archived and will be deleted
+   * immediately. Otherwise, obsolete WALs will be archived prior to deletion.
+   *
+   * When `WAL_size_limit_MB` is nonzero, archived WALs starting with the
+   * earliest will be deleted until the total size of the archive falls below
+   * this limit. All empty WALs will be deleted.
+   *
+   * When `WAL_ttl_seconds` is nonzero, archived WALs older than
+   * `WAL_ttl_seconds` will be deleted.
+   *
+   * When only `WAL_ttl_seconds` is nonzero, the frequency at which archived
+   * WALs are deleted is every `WAL_ttl_seconds / 2` seconds. When only
+   * `WAL_size_limit_MB` is nonzero, the deletion frequency is every ten
+   * minutes. When both are nonzero, the deletion frequency is the minimum of
+   * those two values.
    *
    * @param walTtlSeconds the ttl seconds
    * @return the instance of the current object.
@@ -638,21 +641,24 @@ public interface DBOptionsInterface<T extends DBOptionsInterface<T>> {
   T setWalTtlSeconds(long walTtlSeconds);
 
   /**
-   * WalTtlSeconds() and walSizeLimitMB() affect how archived logs
-   * will be deleted.
-   * <ol>
-   * <li>If both set to 0, logs will be deleted asap and will not get into
-   * the archive.</li>
-   * <li>If WAL_ttl_seconds is 0 and WAL_size_limit_MB is not 0,
-   * WAL files will be checked every 10 min and if total size is greater
-   * then WAL_size_limit_MB, they will be deleted starting with the
-   * earliest until size_limit is met. All empty files will be deleted.</li>
-   * <li>If WAL_ttl_seconds is not 0 and WAL_size_limit_MB is 0, then
-   * WAL files will be checked every WAL_ttl_seconds / 2 and those that
-   * are older than WAL_ttl_seconds will be deleted.</li>
-   * <li>If both are not 0, WAL files will be checked every 10 min and both
-   * checks will be performed with ttl being first.</li>
-   * </ol>
+   * WalTtlSeconds() and walSizeLimitMB() affect when WALs will be archived and
+   * deleted.
+   *
+   * When both are zero, obsolete WALs will not be archived and will be deleted
+   * immediately. Otherwise, obsolete WALs will be archived prior to deletion.
+   *
+   * When `WAL_size_limit_MB` is nonzero, archived WALs starting with the
+   * earliest will be deleted until the total size of the archive falls below
+   * this limit. All empty WALs will be deleted.
+   *
+   * When `WAL_ttl_seconds` is nonzero, archived WALs older than
+   * `WAL_ttl_seconds` will be deleted.
+   *
+   * When only `WAL_ttl_seconds` is nonzero, the frequency at which archived
+   * WALs are deleted is every `WAL_ttl_seconds / 2` seconds. When only
+   * `WAL_size_limit_MB` is nonzero, the deletion frequency is every ten
+   * minutes. When both are nonzero, the deletion frequency is the minimum of
+   * those two values.
    *
    * @return the wal-ttl seconds
    * @see #walSizeLimitMB()
@@ -662,19 +668,22 @@ public interface DBOptionsInterface<T extends DBOptionsInterface<T>> {
   /**
    * WalTtlSeconds() and walSizeLimitMB() affect how archived logs
    * will be deleted.
-   * <ol>
-   * <li>If both set to 0, logs will be deleted asap and will not get into
-   *    the archive.</li>
-   * <li>If WAL_ttl_seconds is 0 and WAL_size_limit_MB is not 0,
-   *    WAL files will be checked every 10 min and if total size is greater
-   *    then WAL_size_limit_MB, they will be deleted starting with the
-   *    earliest until size_limit is met. All empty files will be deleted.</li>
-   * <li>If WAL_ttl_seconds is not 0 and WAL_size_limit_MB is 0, then
-   *    WAL files will be checked every WAL_ttl_secondsi / 2 and those that
-   *    are older than WAL_ttl_seconds will be deleted.</li>
-   * <li>If both are not 0, WAL files will be checked every 10 min and both
-   *    checks will be performed with ttl being first.</li>
-   * </ol>
+   *
+   * When both are zero, obsolete WALs will not be archived and will be deleted
+   * immediately. Otherwise, obsolete WALs will be archived prior to deletion.
+   *
+   * When `WAL_size_limit_MB` is nonzero, archived WALs starting with the
+   * earliest will be deleted until the total size of the archive falls below
+   * this limit. All empty WALs will be deleted.
+   *
+   * When `WAL_ttl_seconds` is nonzero, archived WALs older than
+   * `WAL_ttl_seconds` will be deleted.
+   *
+   * When only `WAL_ttl_seconds` is nonzero, the frequency at which archived
+   * WALs are deleted is every `WAL_ttl_seconds / 2` seconds. When only
+   * `WAL_size_limit_MB` is nonzero, the deletion frequency is every ten
+   * minutes. When both are nonzero, the deletion frequency is the minimum of
+   * those two values.
    *
    * @param sizeLimitMB size limit in mega-bytes.
    * @return the instance of the current object.
@@ -683,21 +692,25 @@ public interface DBOptionsInterface<T extends DBOptionsInterface<T>> {
   T setWalSizeLimitMB(long sizeLimitMB);
 
   /**
-   * {@link #walTtlSeconds()} and {@code #walSizeLimitMB()} affect how archived logs
-   * will be deleted.
-   * <ol>
-   * <li>If both set to 0, logs will be deleted asap and will not get into
-   *    the archive.</li>
-   * <li>If WAL_ttl_seconds is 0 and WAL_size_limit_MB is not 0,
-   *    WAL files will be checked every 10 min and if total size is greater
-   *    then WAL_size_limit_MB, they will be deleted starting with the
-   *    earliest until size_limit is met. All empty files will be deleted.</li>
-   * <li>If WAL_ttl_seconds is not 0 and WAL_size_limit_MB is 0, then
-   *    WAL files will be checked every WAL_ttl_seconds i / 2 and those that
-   *    are older than WAL_ttl_seconds will be deleted.</li>
-   * <li>If both are not 0, WAL files will be checked every 10 min and both
-   *    checks will be performed with ttl being first.</li>
-   * </ol>
+   * WalTtlSeconds() and walSizeLimitMB() affect when WALs will be archived and
+   * deleted.
+   *
+   * When both are zero, obsolete WALs will not be archived and will be deleted
+   * immediately. Otherwise, obsolete WALs will be archived prior to deletion.
+   *
+   * When `WAL_size_limit_MB` is nonzero, archived WALs starting with the
+   * earliest will be deleted until the total size of the archive falls below
+   * this limit. All empty WALs will be deleted.
+   *
+   * When `WAL_ttl_seconds` is nonzero, archived WALs older than
+   * `WAL_ttl_seconds` will be deleted.
+   *
+   * When only `WAL_ttl_seconds` is nonzero, the frequency at which archived
+   * WALs are deleted is every `WAL_ttl_seconds / 2` seconds. When only
+   * `WAL_size_limit_MB` is nonzero, the deletion frequency is every ten
+   * minutes. When both are nonzero, the deletion frequency is the minimum of
+   * those two values.
+   *
    * @return size limit in mega-bytes.
    * @see #walSizeLimitMB()
    */

unreleased_history/behavior_changes/WAL_ttl_seconds_expiration.md

@@ -0,0 +1 @@
+* When `WAL_ttl_seconds > 0`, we now process archived WALs for deletion at least every `WAL_ttl_seconds / 2` seconds. Previously it could be less frequent in case of small `WAL_ttl_seconds` values when size-based expiration (`WAL_size_limit_MB > 0 `) was simultaneously enabled.