Skip to content

Commit

Permalink
Rehash but don't grow when full of tombstones.
Browse files Browse the repository at this point in the history
This problem was found and fixed by José Fonseca in March 2011 for
SmallPtrSet, committed r128566.  But as far as I can tell, all other
llvm hash tables retain the same problem:  the bucket count can grow
without bound while size() remains near constant by repeated
insert/erase cycles that tend to fill the container with tombstones. 
Here is a demo that has been reduced to a trivial case:

int
main()
{
   llvm::DenseSet<unsigned> d;
   for (unsigned i = 0; i < 0xFFFFFFF; ++i)
   {
       d.insert(i);
       d.erase(i);
   }
}

While the container size() never grows above 1, the bucket count grows
like this:

nb = 64
nb = 128
nb = 256
nb = 512
nb = 1024
nb = 2048
nb = 4096
nb = 8192
nb = 16384
nb = 32768
nb = 65536
nb = 131072
nb = 262144
nb = 524288
nb = 1048576
nb = 2097152
nb = 4194304
nb = 8388608
nb = 16777216
nb = 33554432
nb = 67108864
nb = 134217728
nb = 268435456

The above program currently consumes a few GB ram.  This patch brings
the memory consumption down by several orders of magnitude, and keeps
the bucket count at 64 for the above test.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@193689 91177308-0d34-0410-b5e6-96231b3b80d8
  • Loading branch information
Howard Hinnant committed Oct 30, 2013
1 parent 6ff1ef9 commit 2aaa47f
Showing 1 changed file with 2 additions and 3 deletions.
5 changes: 2 additions & 3 deletions include/llvm/ADT/DenseMap.h
Original file line number Diff line number Diff line change
Expand Up @@ -438,9 +438,8 @@ class DenseMapBase {
this->grow(NumBuckets * 2);
LookupBucketFor(Key, TheBucket);
NumBuckets = getNumBuckets();
}
if (NumBuckets-(NewNumEntries+getNumTombstones()) <= NumBuckets/8) {
this->grow(NumBuckets * 2);
} else if (NumBuckets-(NewNumEntries+getNumTombstones()) <= NumBuckets/8) {
this->grow(NumBuckets);
LookupBucketFor(Key, TheBucket);
}
assert(TheBucket);
Expand Down

0 comments on commit 2aaa47f

Please sign in to comment.