Skip to content

Commit

Permalink
fix(store): lower number of placement groups
Browse files Browse the repository at this point in the history 
We were incorrectly configuring a large number of placement groups
for the Ceph storage pools, resulting in each store-daemon doing
more work than necessary.

This commit removes the logic to reconfigure the number of placement
groups for the default pools (since they are already larger than 64),
and lowers the default number of placement groups from 128 to 64,
resulting in 768 placement groups per daemon. We also reconfigure
the warning for number of placement groups per OSD to 800 so we
no longer warn in the typical case.
  • Loading branch information
@carmstrong
carmstrong committed May 11, 2015
1 parent d89ea98 commit 6c9765b
Show file tree
Hide file tree
Showing 7 changed files with 18 additions and 47 deletions.
1 change: 1 addition & 0 deletions docs/customizing_deis/store_daemon_settings.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,7 @@ setting description
/deis/store/adminKeyring keyring for an admin user to access the Ceph cluster (set by store-monitor)
/deis/store/fsid Ceph filesystem ID (set by store-monitor)
/deis/store/hosts/* deis-monitor hosts (set by store-monitor)
/deis/store/maxPGsPerOSDWarning threshold for warning on number of placement groups per OSD (set by store-monitor)
/deis/store/monKeyring keyring for the monitor to access the Ceph cluster (set by store-monitor)
/deis/store/monSetupComplete set when the Ceph cluster setup is complete (set by store-monitor)
/deis/store/monSetupLock host of store-monitor that completed setup (set by store-monitor)
Expand Down
2 changes: 1 addition & 1 deletion docs/customizing_deis/store_gateway_settings.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,6 @@ The following etcd keys are set by the store-gateway component, typically in its
================================== =================================================================================
setting description
================================== =================================================================================
/deis/store/defaultPoolsConfigured set once the gateway configures the default storage pools on first boot
/deis/store/gatewayKeyring keyring for the gateway user to access the Ceph cluster (generated by Ceph)
/deis/store/gateway/accessKey S3 API access used to access store-gateway (generated by Ceph)
/deis/store/gateway/host IP address of the host running store-gateway
Expand All @@ -40,6 +39,7 @@ setting description
/deis/store/adminKeyring keyring for an admin user to access the Ceph cluster (set by store-monitor)
/deis/store/fsid Ceph filesystem ID (set by store-monitor)
/deis/store/hosts/* deis-monitor hosts (set by store-monitor)
/deis/store/maxPGsPerOSDWarning threshold for warning on number of placement groups per OSD (set by store-monitor)
/deis/store/monKeyring keyring for the monitor to access the Ceph cluster (set by store-monitor)
/deis/store/monSetupComplete set when the Ceph cluster setup is complete (set by store-monitor)
/deis/store/monSetupLock host of store-monitor that completed setup (set by store-monitor)
Expand Down
1 change: 1 addition & 0 deletions docs/customizing_deis/store_metadata_settings.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,7 @@ setting description
/deis/store/adminKeyring keyring for an admin user to access the Ceph cluster (set by store-monitor)
/deis/store/fsid Ceph filesystem ID (set by store-monitor)
/deis/store/hosts/* deis-monitor hosts (set by store-monitor)
/deis/store/maxPGsPerOSDWarning threshold for warning on number of placement groups per OSD (set by store-monitor)
/deis/store/monKeyring keyring for the monitor to access the Ceph cluster (set by store-monitor)
/deis/store/monSetupComplete set when the Ceph cluster setup is complete (set by store-monitor)
/deis/store/monSetupLock host of store-monitor that completed setup (set by store-monitor)
Expand Down
7 changes: 4 additions & 3 deletions docs/customizing_deis/store_monitor_settings.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -19,19 +19,20 @@ Settings set by store-monitor
-----------------------------
The following etcd keys are set by the store-monitor component, typically in its /bin/boot script.

============================= =================================================================================
=============================== ==================================================================================
setting description
============================= =================================================================================
=============================== ==================================================================================
/deis/store/adminKeyring keyring for an admin user to access the Ceph cluster
/deis/store/fsid Ceph filesystem ID
/deis/store/hosts/$HOST hostname (not IP) of the host running this store-monitor instance
/deis/store/maxPGsPerOSDWarning threshold for warning on number of placement groups per OSD (set by store-monitor)
/deis/store/monKeyring keyring for the monitor to access the Ceph cluster
/deis/store/monSetupComplete set when the Ceph cluster setup is complete
/deis/store/monSetupLock IP address of the monitor instance that is or has set up the Ceph cluster
/deis/store/minSize minimum number of store-daemons necessary for the cluster to accept writes
/deis/store/pgNum number of Ceph placement groups for the storage pools
/deis/store/size number of replicas for data stored in Ceph
============================= =================================================================================
=============================== ==================================================================================

Settings used by store-monitor
------------------------------
Expand Down
1 change: 1 addition & 0 deletions store/base/templates/ceph.conf
View file
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@ fsid = {{ getv "/deis/store/fsid" }}
mon initial members = {{ getv "/deis/store/monSetupLock" }}
mon host = {{ join (getvs "/deis/store/hosts/*") "," }}
mon addr = {{ range $index, $element := (gets "/deis/store/hosts/*") }}{{if $index}},{{end}}{{ base $element.Key }}:6789{{ end }}
mon pg warn max per osd = {{ getv "/deis/store/maxPGsPerOSDWarning" }}
auth cluster required = cephx
auth service required = cephx
auth client required = cephx
Expand Down
39 changes: 0 additions & 39 deletions store/gateway/bin/boot
View file
Original file line number Diff line number Diff line change
Expand Up @@ -31,45 +31,6 @@ until confd -onetime -node $ETCD --confdir /app --log-level error; do
sleep $(($ETCD_TTL/2)) # sleep for half the TTL
done

# set the number of placement groups for the default pools - they come up with defaults that are too low
if ! etcdctl --no-sync -C $ETCD get /deis/store/defaultPoolsConfigured >/dev/null 2>&1 ; then
echo "store-gateway: setting pg_num values for default pools..."
function set_until_success {
set +e

echo "store-gateway: checking pool $1..."
if ! ceph osd pool get $1 pg_num | grep "pg_num: $2" ; then
ceph osd pool set $1 pg_num $2 2>/dev/null
PG_SET=$?
until [[ $PG_SET -eq 0 ]]; do
sleep 5
ceph osd pool set $1 pg_num $2 2>/dev/null
PG_SET=$?
done
fi

if ! ceph osd pool get $1 pgp_num | grep "pgp_num: $2" ; then
ceph osd pool set $1 pgp_num $2 2>/dev/null
PGP_SET=$?
until [[ $PGP_SET -eq 0 ]]; do
sleep 5
ceph osd pool set $1 pgp_num $2 2>/dev/null
PGP_SET=$?
done
fi

set -e
}

PG_NUM=`etcdctl --no-sync -C $ETCD get /deis/store/pgNum`

set_until_success data ${PG_NUM}
set_until_success rbd ${PG_NUM}
set_until_success metadata ${PG_NUM}

etcdctl --no-sync -C $ETCD set /deis/store/defaultPoolsConfigured youBetcha >/dev/null
fi

# we generate a key for the gateway. we can do this because we have the client key templated out
if ! etcdctl --no-sync -C $ETCD get /deis/store/gatewayKeyring >/dev/null 2>&1 ; then
ceph-authtool --create-keyring /etc/ceph/ceph.client.radosgw.keyring
Expand Down
14 changes: 10 additions & 4 deletions store/monitor/bin/boot
View file
Original file line number Diff line number Diff line change
Expand Up @@ -6,10 +6,15 @@ set -e
ETCD_PORT=${ETCD_PORT:-4001}
ETCD="$HOST:$ETCD_PORT"
ETCD_PATH=${ETCD_PATH:-/deis/store}
NUM_STORES=${NUM_STORES:-3}
PG_NUM=${PG_NUM:-128} # default for 3 OSDs
HOSTNAME=`hostname`

# These defaults are for 3 hosts
NUM_STORES=${NUM_STORES:-3}
PG_NUM=${PG_NUM:-64}
## We set this to the number of PGs before re-evaluating the PG count so users upgrading don't see the warning
## Now, 12 pools * 64 pgs per pool = 768 PGs per OSD
PGS_PER_OSD_WARNING=${PGS_PER_OSD_WARNING:-1536}

function etcd_set_default {
set +e
etcdctl --no-sync -C $ETCD mk $ETCD_PATH/$1 $2 >/dev/null 2>&1
Expand All @@ -29,10 +34,11 @@ if ! etcdctl --no-sync -C $ETCD get ${ETCD_PATH}/monSetupComplete >/dev/null 2>&

# set some defaults in etcd if they're not passed in as environment variables
# these are templated in ceph.conf
etcd_set_default size ${NUM_STORES}
etcd_set_default delayStart 15
etcd_set_default maxPGsPerOSDWarning ${PGS_PER_OSD_WARNING}
etcd_set_default minSize 1
etcd_set_default pgNum ${PG_NUM}
etcd_set_default delayStart 15
etcd_set_default size ${NUM_STORES}

# Generate administrator key
ceph-authtool /etc/ceph/ceph.client.admin.keyring --create-keyring --gen-key -n client.admin --set-uid=0 --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow'
Expand Down

0 comments on commit 6c9765b

Please sign in to comment.