Skip to content

Commit

Permalink
Delete empty WAL files on reopen (facebook#11409)
Browse files Browse the repository at this point in the history
Summary:
When a DB is opened, RocksDB creates an empty WAL file. When the DB is reopened and the WAL is empty, the min log number to keep is not advanced until a memtable flush happens. If a process crashes soon after reopening the DB, its likely that no memtable flush would have happened, which means the empty WAL file is not deleted. In a crash loop scenario, this leads to empty WAL files accumulating. Fix this by ensuring the min log number is advanced if the WAL is empty.

Pull Request resolved: facebook#11409

Test Plan: Add a unit test

Reviewed By: ajkr

Differential Revision: D45281685

Pulled By: anand1976

fbshipit-source-id: 0225877c613e65ffb30972a0051db2830105423e
  • Loading branch information
anand1976 authored and facebook-github-bot committed May 2, 2023
1 parent 41a7fbf commit 03a892a
Show file tree
Hide file tree
Showing 3 changed files with 29 additions and 1 deletion.
3 changes: 3 additions & 0 deletions HISTORY.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,9 @@
### Public API Changes
* Add `MakeSharedCache()` construction functions to various cache Options objects, and deprecated the `NewWhateverCache()` functions with long parameter lists.

### Bug Fixes
* Delete an empty WAL file on DB open if the log number is less than the min log number to keep

## 8.2.0 (04/24/2023)
### Public API Changes
* `SstFileWriter::DeleteRange()` now returns `Status::InvalidArgument` if the range's end key comes before its start key according to the user comparator. Previously the behavior was undefined.
Expand Down
2 changes: 1 addition & 1 deletion db/db_impl/db_impl_open.cc
Original file line number Diff line number Diff line change
Expand Up @@ -1453,7 +1453,7 @@ Status DBImpl::RecoverLogFiles(const std::vector<uint64_t>& wal_numbers,
recovery_ctx->UpdateVersionEdits(cfd, iter->second);
}

if (flushed) {
if (flushed || !data_seen) {
VersionEdit wal_deletion;
if (immutable_db_options_.track_and_verify_wals_in_manifest) {
wal_deletion.DeleteWalsBefore(max_wal_number + 1);
Expand Down
25 changes: 25 additions & 0 deletions db/db_wal_test.cc
Original file line number Diff line number Diff line change
Expand Up @@ -2399,6 +2399,31 @@ TEST_F(DBWALTest, GetCompressedWalsAfterSync) {
Status s = dbfull()->GetSortedWalFiles(wals);
ASSERT_OK(s);
}

TEST_F(DBWALTest, EmptyWalReopenTest) {
Options options = CurrentOptions();
options.env = env_;
CreateAndReopenWithCF({"pikachu"}, options);

// make sure we can re-open it.
ASSERT_OK(TryReopenWithColumnFamilies({"default", "pikachu"}, options));

{
std::vector<std::string> files;
int num_wal_files = 0;
ASSERT_OK(env_->GetChildren(dbname_, &files));
for (const auto& file : files) {
uint64_t number = 0;
FileType type = kWalFile;
if (ParseFileName(file, &number, &type) && type == kWalFile) {
num_wal_files++;
}
}

ASSERT_EQ(num_wal_files, 1);
}
}

} // namespace ROCKSDB_NAMESPACE

int main(int argc, char** argv) {
Expand Down

0 comments on commit 03a892a

Please sign in to comment.