By default, auto configure the size of Bookie read/write cache (apach…

…e#4297)

* By default, auto configure the size of Bookie read/write cache

* Updated ref docs

conf/bookkeeper.conf

@@ -628,12 +628,15 @@ httpServerClass=org.apache.bookkeeper.http.vertx.VertxHttpServer
 
 # Size of Write Cache. Memory is allocated from JVM direct memory.
 # Write cache is used to buffer entries before flushing into the entry log
-# For good performance, it should be big enough to hold a sub
-dbStorage_writeCacheMaxSizeMb=512
+# For good performance, it should be big enough to hold a substantial amount
+# of entries in the flush interval
+#  By default it will be allocated to 1/4th of the available direct memory
+dbStorage_writeCacheMaxSizeMb=
 
 # Size of Read cache. Memory is allocated from JVM direct memory.
 # This read cache is pre-filled doing read-ahead whenever a cache miss happens
-dbStorage_readAheadCacheMaxSizeMb=256
+#  By default it will be allocated to 1/4th of the available direct memory
+dbStorage_readAheadCacheMaxSizeMb=
 
 # How many entries to pre-fill in cache after a read cache miss
 dbStorage_readAheadCacheBatchSize=1000
@@ -645,8 +648,8 @@ dbStorage_readAheadCacheBatchSize=1000
 # Size of RocksDB block-cache. For best performance, this cache
 # should be big enough to hold a significant portion of the index
 # database which can reach ~2GB in some cases
-# Default is 256 MBytes
-dbStorage_rocksDB_blockCacheSize=268435456
+# Default is to use 10% of the direct memory size
+dbStorage_rocksDB_blockCacheSize=
 
 # Other RocksDB specific tunables
 dbStorage_rocksDB_writeBufferSizeMB=64

conf/standalone.conf

@@ -477,12 +477,15 @@ ledgerStorageClass=org.apache.bookkeeper.bookie.storage.ldb.DbLedgerStorage
 
 # Size of Write Cache. Memory is allocated from JVM direct memory.
 # Write cache is used to buffer entries before flushing into the entry log
-# For good performance, it should be big enough to hold a sub
-dbStorage_writeCacheMaxSizeMb=256
+# For good performance, it should be big enough to hold a substantial amount
+# of entries in the flush interval
+# By default it will be allocated to 1/4th of the available direct memory
+dbStorage_writeCacheMaxSizeMb=
 
 # Size of Read cache. Memory is allocated from JVM direct memory.
 # This read cache is pre-filled doing read-ahead whenever a cache miss happens
-dbStorage_readAheadCacheMaxSizeMb=64
+# By default it will be allocated to 1/4th of the available direct memory
+dbStorage_readAheadCacheMaxSizeMb=
 
 # How many entries to pre-fill in cache after a read cache miss
 dbStorage_readAheadCacheBatchSize=1000
@@ -496,8 +499,8 @@ flushInterval=60000
 # Size of RocksDB block-cache. For best performance, this cache
 # should be big enough to hold a significant portion of the index
 # database which can reach ~2GB in some cases
-# Default is 16 MBytes
-dbStorage_rocksDB_blockCacheSize=16777216
+# Default is to use 10% of the direct memory size
+dbStorage_rocksDB_blockCacheSize=
 
 # Other RocksDB specific tunables
 dbStorage_rocksDB_writeBufferSizeMB=4

site2/docs/reference-configuration.md

@@ -84,10 +84,10 @@ BookKeeper is a replicated log storage system that Pulsar uses for durable stora
 |useHostNameAsBookieID|Whether the bookie should use its hostname to register with the coordination service (e.g.: zookeeper service). When false, bookie will use its ipaddress for the registration.|false|
 |statsProviderClass||org.apache.bookkeeper.stats.prometheus.PrometheusMetricsProvider|
 |prometheusStatsHttpPort||8000|
-|dbStorage_writeCacheMaxSizeMb|Size of Write Cache. Memory is allocated from JVM direct memory. Write cache is used to buffer entries before flushing into the entry log For good performance, it should be big enough to hold a sub|512|
-|dbStorage_readAheadCacheMaxSizeMb|Size of Read cache. Memory is allocated from JVM direct memory. This read cache is pre-filled doing read-ahead whenever a cache miss happens|256|
+|dbStorage_writeCacheMaxSizeMb|Size of Write Cache. Memory is allocated from JVM direct memory. Write cache is used to buffer entries before flushing into the entry log For good performance, it should be big enough to hold a sub|25% of direct memory|
+|dbStorage_readAheadCacheMaxSizeMb|Size of Read cache. Memory is allocated from JVM direct memory. This read cache is pre-filled doing read-ahead whenever a cache miss happens|25% of direct memory|
 |dbStorage_readAheadCacheBatchSize|How many entries to pre-fill in cache after a read cache miss|1000|
-|dbStorage_rocksDB_blockCacheSize|Size of RocksDB block-cache. For best performance, this cache should be big enough to hold a significant portion of the index database which can reach ~2GB in some cases|268435456|
+|dbStorage_rocksDB_blockCacheSize|Size of RocksDB block-cache. For best performance, this cache should be big enough to hold a significant portion of the index database which can reach ~2GB in some cases|10% of direct memory|
 |dbStorage_rocksDB_writeBufferSizeMB||64|
 |dbStorage_rocksDB_sstSizeInMB||64|
 |dbStorage_rocksDB_blockSize||65536|