PG数计算

PG数计算

原地址：http://xiaqunfeng.cc/2017/09/15/too-many-PGs-per-OSD/

ceph告警问题：”too many PGs per OSD” 的解决方法，以及pg数量的合理设定

现象

1 2 3 4 5 6 7 8 9 10 11 12 13

# ceph -s cluster 4c7ec5af-cbd3-40fd-8c96-0615c77660d4 health HEALTH_WARN too many PGs per OSD (412 > max 300) monmap e2: 3 mons at {ceph0=172.21.1.21:6789/0,ceph1=172.21.1.22:6789/0,ceph2=172.21.1.23:6789/0} election epoch 1780, quorum 0,1,2 ceph0,ceph1,ceph2 mgr active: ceph0 standbys: ceph1, ceph2 osdmap e94: 6 osds: 6 up, 6 in flags sortbitwise,require_jewel_osds,require_kraken_osds pgmap v161317: 824 pgs, 10 pools, 30201 MB data, 8642 objects 90831 MB used, 181 GB / 269 GB avail 824 active+clean client io 34800 B/s wr, 0 op/s rd, 9 op/s wr

原因

• 集群osd 数量较少
• 搭建rgw网关、OpenStack、容器组件等，pool创建较多，每个pool默认需要占用一些pg，pool中pg数目设置不合理，导致集群 total pg 数过多

解决

方法

调整每个osd默认pg数，参数为 mon_pg_warn_max_per_osd ，当前默认参数如下：

1 2	# ceph --show-config | grep mon_pg_warn_max_per_osd mon_pg_warn_max_per_osd = 300

步骤

1、修改ceph配置文件

1 2 3 4 5

# cd /etc/ceph # vim ceph.conf [global] ....... mon_pg_warn_max_per_osd = 500

2、将配置文件推到mon所在的其他节点

1	# ceph-deploy --overwrite-conf config push ceph1 ceph2

3、重启mon进程

1	# systemctl restart ceph-mon.target

重启成功后，再次查看配置项

1 2	# ceph --show-config | grep mon_pg_warn_max_per_osd mon_pg_warn_max_per_osd = 500

此时集群状态ok

1 2 3 4 5 6 7 8 9 10 11 12

# ceph -s cluster 4c7ec5af-cbd3-40fd-8c96-0615c77660d4 health HEALTH_OK monmap e2: 3 mons at {ceph0=172.21.1.21:6789/0,ceph1=172.21.1.22:6789/0,ceph2=172.21.1.23:6789/0} election epoch 1780, quorum 0,1,2 ceph0,ceph1,ceph2 mgr active: ceph0 standbys: ceph1, ceph2 osdmap e94: 6 osds: 6 up, 6 in flags sortbitwise,require_jewel_osds,require_kraken_osds pgmap v161317: 824 pgs, 10 pools, 30201 MB data, 8642 objects 90831 MB used, 181 GB / 269 GB avail 824 active+clean client io 34800 B/s wr, 0 op/s rd, 9 op/s wr

pg数目的设定

信息查询

1、查看当前osd数目

1 2	# ceph osd ls | wc -l 6

2、查看当前有多少个pool

1 2	# ceph osd pool ls | wc -l 10

3、查看 replication pool 的数量

1 2 3 4 5 6 7 8 9 10 11

# ceph osd dump | grep repli pool 0 'rbd' replicated size 3 min_size 2 crush_ruleset 0 object_hash rjenkins pg_num 256 pgp_num 256 last_change 31 flags hashpspool stripe_width 0 pool 1 '.rgw.root' replicated size 3 min_size 2 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 14 flags hashpspool stripe_width 0 pool 2 'default.rgw.control' replicated size 3 min_size 2 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 16 flags hashpspool stripe_width 0 pool 3 'default.rgw.data.root' replicated size 3 min_size 2 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 19 flags hashpspool stripe_width 0 pool 4 'default.rgw.gc' replicated size 3 min_size 2 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 20 flags hashpspool stripe_width 0 pool 5 'default.rgw.lc' replicated size 3 min_size 2 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 21 flags hashpspool stripe_width 0 pool 6 'default.rgw.log' replicated size 3 min_size 2 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 23 flags hashpspool stripe_width 0 pool 7 'default.rgw.users.uid' replicated size 3 min_size 2 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 26 flags hashpspool stripe_width 0 pool 8 'kube' replicated size 3 min_size 2 crush_ruleset 0 object_hash rjenkins pg_num 256 pgp_num 256 last_change 36 flags hashpspool stripe_width 0 pool 9 'stage' replicated size 3 min_size 2 crush_ruleset 0 object_hash rjenkins pg_num 256 pgp_num 256 last_change 54 flags hashpspool stripe_width 0

可以看到，每个pool都是 3 副本（replicated size 3）

total pg num

公式如下：

1	Total PGs = (Total_number_of_OSD * 100) / max_replication_count

结果必须取最接近该数的 2 的幂

比如，根据以上信息：

1 2 3

Total_number_of_OSD = 6 max_replication_count = 3 Total PGs = 200

最接近 200 的 2 的幂是 256。所以推荐的集群最大 total PGs 数为 256。

pool pg num

每个 pool 的 pg 数目计算：

1	Total PGs = ((Total_number_of_OSD * 100) / max_replication_count) / pool_count

结果同样要取最接近的 2 的幂。

对应该例，每个 pool 的 pg num 为：

1 2	pool_count = 10 Total PGs = 200 / 10 = 20

所以每个 pool 的平均分配 pg num 为 16。

pg num command

得到和设置指定 pool 中的 pg_num 和 pgp_num

command	function
ceph osd pool create <pool-name> <pg-number> <pgp-number>	To create a new pool
ceph osd pool get <pool-name> <pg_num>	To get number of PG in a pool
ceph osd pool get <pool-name> <pgp_num>	To get number of PGP in a pool
ceph osd pool set <pool-name> <pg_num number>	To increase number of PG in a pool
ceph osd pool set <pool-name> <pgp_num number>	To increase number of PGP in a pool

创建pool时如果不指定 pg_num，默认为8

1 2	# ceph --show-config | grep osd_pool_default_pg_num osd_pool_default_pg_num = 8