012 Ceph多区域网关
一、基本概念
1.1 多区域概念
Multi-Size功能是从J版本开始的。一个single zone配置通常由一个zone group组成，该zone group包含一个zone和多个用于负载均衡的RGW实例。从K版本开始，ceph为RGW提供了Multi-Size的配置选项
1.2 多区域相关术语
区域（zone）: 一个ceph集群可以包含多个区域，一个区域只属于一个集群，一个区域可以有多个RGW
区域组（zonegroup）：由一个或多个区域组成，包含一个主区域（master zone），其他区域称为Secondary Zone，区域组内的所有区域之间同步数据
域（realm）: 同一个或多个区域组组成，包含一个主区域组，其他都次区域组。域中的所有rados网关都从位于主区域组和主区域中的rados网关拉取配置
注意： master zone group中的master zone处理所元数据更新，因此创建用户、bucket等操作都必须经由master zone
1.3 多区域网关配置架构
single-zone：一个realm中只有一个zonegroup和一个zone，可以有多个RGW
multi-zone：一个relam中只有一个zonegroup，但是有多个zone。一个realm中存储的数据复制到该zonegroup中的所有zone中
multi-zonegroup：一个realm中有多个zonegroup，每个zonegroup中又有一个或多个zone
multi-realms：多个realm
两个集群配置两个zone，实现两个zone之间的数据同步：
periods和Epochs：
每个zone有关联的period，每个period有关联的epoch。
period用于跟踪zone、zone group和realm的配置状态
epoch用于跟踪zone period的配置变更的版本号
每个period具有唯一ID，含有zone配置，并且知道其先前period的id
在更改zone的配置时，需要更新zone的当前period
1.4 多区域同步流程
RGW在所有zone group集合之间同步元数据和数据操作。元数据操作与bucket相关：创建、删除、启用和禁用版本控制、管理用户。meta master位于master zone group中的master zone，负责管理元数据更新
多区域配置后处于活跃状态时，RGW会在master和secondary区域之间执行一次初始的完整同步。随后的更新是增量更新
当RGW将数据写入zone group的任意zone时，它会在该zone group的所有其他zone之间同步这一数据
当RGW同步数据时，所有活跃的网关会更新数据日志并通知其他网关
当RGW网关因用户操作而同步元数据时，主网关会更新元数据日志并通知其他RGW网关
两个集群配置两个zone，实现两个zone之间的数据同步：periods和Epochs：每个zone有关联的period，每个period有关联的epoch。period用于跟踪zone、zone group和realm的配置状态epoch用于跟踪zone period的配置变更的版本号每个period具有唯一ID，含有zone配置，并且知道其先前period的id在更改zone的配置时，需要更新zone的
当前period多区域同步流程：RGW在所有zone group集合之间同步元数据和数据操作。元数据操作与bucket相关：创建、删除、启用和禁用版本控制、管理用户。meta master位于master zone group中的master zone，负责管理元数据更新多区域配置后处于活跃状态时，RGW会在master和secondary区域之间执行一次初始的完整同步。随后
的更新是增量更新当RGW将数据写入zone group的任意zone时，它会在该zone group的所有其他zone之间同步这一数据当RGW同步数据时，所有活跃的网关会更新数据日志并通知其他网关当RGW网关因用户操作而同步元数据时，主网关会更新元数据日志并通知其他RGW网关
二、配置
2.1 查看
[root@ceph5 ~]# radosgw-admin realm list
[root@ceph5 ~]# radosgw-admin zonegroup list
[root@ceph5 ~]# radosgw-admin zone list
[root@ceph5 ~]# radosgw-admin zone list --rgw-zonegroup default
2.2 创建realm
[root@ceph5 ~]# radosgw-admin realm create --rgw-realm hubei --default
2.3 创建zonegroup
[root@ceph5 ~]# radosgw-admin zonegroup create --rgw-realm hubei --rgw-zonegroup fancheng --default --master
2.4 修改名
[root@ceph5 ~]# radosgw-admin zonegroup rename --rgw-realm hubei --rgw-zonegroup fancheng --zonegroup-new-name xiangyang
[root@ceph5 ~]# radosgw-admin zonegroup get --rgw-zonegroup xiangyang
2.5 删除一个zonegroup
[root@ceph5 ~]# radosgw-admin realm remove --rgw-realm hubei --rgw-zonegroup xiangyang
[root@ceph5 ~]# radosgw-admin zonegroup list
[root@ceph5 ~]# radosgw-admin zonegroup delete --rgw-zonegroup xiangyang
[root@ceph5 ~]# radosgw-admin zonegroup list
2.6 重建一个zonegroup
[root@ceph5 ~]# radosgw-admin zonegroup create --rgw-realm hubei --rgw-zonegroup xiangyang --default --master
{
"id": "5acbb712-0f7f-4108-8d93-ea75a19e33b3",
"name": "xiangyang",
"api_name": "xiangyang",
"is_master": "true",
"endpoints": [],
"hostnames": [],
"hostnames_s3website": [],
"master_zone": "",
"zones": [],
"placement_targets": [],
"default_placement": "",
"realm_id": "d4668fc2-ceed-4eb2-a5e7-a70c2aa7deba"
}
2.7 创建一个zone
[root@ceph5 ~]# radosgw-admin zone create --rgw-zonegroup xiangyang --rgw-zone fancheng --default --endpoints http://ceph5.lab.example.com
{
"id": "b337a562-5921-46fc-aad2-e70e99454e5f",
"name": "fancheng",
"domain_root": "fancheng.rgw.meta:root",
"control_pool": "fancheng.rgw.control",
"gc_pool": "fancheng.rgw.log:gc",
"lc_pool": "fancheng.rgw.log:lc",
"log_pool": "fancheng.rgw.log",
"intent_log_pool": "fancheng.rgw.log:intent",
"usage_log_pool": "fancheng.rgw.log:usage",
"reshard_pool": "fancheng.rgw.log:reshard",
"user_keys_pool": "fancheng.rgw.meta:users.keys",
"user_email_pool": "fancheng.rgw.meta:users.email",
"user_swift_pool": "fancheng.rgw.meta:users.swift",
"user_uid_pool": "fancheng.rgw.meta:users.uid",
"system_key": {
"access_key": "",
"secret_key": ""
},
"placement_pools": [
{
"key": "default-placement",
"val": {
"index_pool": "fancheng.rgw.buckets.index",
"data_pool": "fancheng.rgw.buckets.data",
"data_extra_pool": "fancheng.rgw.buckets.non-ec",
"index_type": ,
"compression": ""
}
}
],
"metadata_heap": "",
"tier_config": [],
"realm_id": "d4668fc2-ceed-4eb2-a5e7-a70c2aa7deba"
}
2.8 为zone创建一个系统账号
[root@ceph5 ~]# radosgw-admin user create --uid syncuser --display-name "sync user" --system
{
"user_id": "syncuser",
"display_name": "sync user",
"email": "",
"suspended": ,
"max_buckets": ,
"auid": ,
"subusers": [],
"keys": [
{
"user": "syncuser",
"access_key": "42Y4FBWY5VDCRLG9GDM8",
"secret_key": "BiGRHcqL03RFoFWAbymFZj6xDQAXYFVb60cIzvav"
}
],
"swift_keys": [],
"caps": [],
"op_mask": "read, write, delete",
"system": "true",
"default_placement": "",
"placement_tags": [],
"bucket_quota": {
"enabled": false,
"check_on_raw": false,
"max_size": -,
"max_size_kb": ,
"max_objects": -
},
"user_quota": {
"enabled": false,
"check_on_raw": false,
"max_size": -,
"max_size_kb": ,
"max_objects": -
},
"temp_url_keys": [],
"type": "rgw"
}
2.9 修改zone绑定这个用户
[root@ceph5 ~]# radosgw-admin zone modify --rgw-zonegroup xiangyang --rgw-zone fancheng --access 42Y4FBWY5VDCRLG9GDM8 --secret BiGRHcqL03RFoFWAbymFZj6xDQAXYFVb60cIzvav
{
"id": "b337a562-5921-46fc-aad2-e70e99454e5f",
"name": "fancheng",
"domain_root": "fancheng.rgw.meta:root",
"control_pool": "fancheng.rgw.control",
"gc_pool": "fancheng.rgw.log:gc",
"lc_pool": "fancheng.rgw.log:lc",
"log_pool": "fancheng.rgw.log",
"intent_log_pool": "fancheng.rgw.log:intent",
"usage_log_pool": "fancheng.rgw.log:usage",
"reshard_pool": "fancheng.rgw.log:reshard",
"user_keys_pool": "fancheng.rgw.meta:users.keys",
"user_email_pool": "fancheng.rgw.meta:users.email",
"user_swift_pool": "fancheng.rgw.meta:users.swift",
"user_uid_pool": "fancheng.rgw.meta:users.uid",
"system_key": {
"access_key": "",
"secret_key": "BiGRHcqL03RFoFWAbymFZj6xDQAXYFVb60cIzvav"
},
"placement_pools": [
{
"key": "default-placement",
"val": {
"index_pool": "fancheng.rgw.buckets.index",
"data_pool": "fancheng.rgw.buckets.data",
"data_extra_pool": "fancheng.rgw.buckets.non-ec",
"index_type": ,
"compression": ""
}
}
],
"metadata_heap": "",
"tier_config": [],
"realm_id": "d4668fc2-ceed-4eb2-a5e7-a70c2aa7deba"
}
[root@ceph5 ~]# radosgw-admin zonegroup get --rgw-zonegroup xiangyang
{
"id": "5acbb712-0f7f-4108-8d93-ea75a19e33b3",
"name": "xiangyang",
"api_name": "xiangyang",
"is_master": "true",
"endpoints": [],
"hostnames": [],
"hostnames_s3website": [],
"master_zone": "b337a562-5921-46fc-aad2-e70e99454e5f",
"zones": [
{
"id": "b337a562-5921-46fc-aad2-e70e99454e5f",
"name": "fancheng",
"endpoints": [
"http://ceph5.lab.example.com"
],
"log_meta": "false",
"log_data": "false",
"bucket_index_max_shards": ,
"read_only": "false",
"tier_type": "",
"sync_from_all": "true",
"sync_from": []
}
],
"placement_targets": [
{
"name": "default-placement",
"tags": []
}
],
"default_placement": "default-placement",
"realm_id": "d4668fc2-ceed-4eb2-a5e7-a70c2aa7deba"
}
2.10 查看用户
[root@ceph5 ~]# radosgw-admin user list
[root@ceph5 ~]# radosgw-admin user list --rgw-zonegroup default --rgw-zone default
2.11 实施刷盘操作,更新period
[root@ceph5 ~]# radosgw-admin period update --commit
{
"id": "1c6ccdef-ca02-44b5-b212-ba68acbd6aad",
"epoch": ,
"predecessor_uuid": "55fafff6-9c4c-4d54-8801-ad48861221a1",
"sync_status": [],
"period_map": {
"id": "1c6ccdef-ca02-44b5-b212-ba68acbd6aad",
"zonegroups": [
{
"id": "5acbb712-0f7f-4108-8d93-ea75a19e33b3",
"name": "xiangyang",
"api_name": "xiangyang",
"is_master": "true",
"endpoints": [],
"hostnames": [],
"hostnames_s3website": [],
"master_zone": "b337a562-5921-46fc-aad2-e70e99454e5f",
"zones": [
{
"id": "b337a562-5921-46fc-aad2-e70e99454e5f",
"name": "fancheng",
"endpoints": [
"http://ceph5.lab.example.com"
],
"log_meta": "false",
"log_data": "false",
"bucket_index_max_shards": ,
"read_only": "false",
"tier_type": "",
"sync_from_all": "true",
"sync_from": []
}
],
"placement_targets": [
{
"name": "default-placement",
"tags": []
}
],
"default_placement": "default-placement",
"realm_id": "d4668fc2-ceed-4eb2-a5e7-a70c2aa7deba"
}
],
"short_zone_ids": [
{
"key": "b337a562-5921-46fc-aad2-e70e99454e5f",
"val":
}
]
},
"master_zonegroup": "5acbb712-0f7f-4108-8d93-ea75a19e33b3",
"master_zone": "b337a562-5921-46fc-aad2-e70e99454e5f",
"period_config": {
"bucket_quota": {
"enabled": false,
"check_on_raw": false,
"max_size": -,
"max_size_kb": ,
"max_objects": -
},
"user_quota": {
"enabled": false,
"check_on_raw": false,
"max_size": -,
"max_size_kb": ,
"max_objects": -
}
},
"realm_id": "d4668fc2-ceed-4eb2-a5e7-a70c2aa7deba",
"realm_name": "hubei",
"realm_epoch":
}
[root@ceph5 ~]# ceph osd pool ls
2.12 更新配置文件并重启
[root@ceph5 ~]# vim /etc/ceph/backup.conf
[root@ceph5 ~]# systemctl restart ceph-radosgw@rgw.ceph5
为当前zone创建用户
[root@ceph5 ~]# radosgw-admin user create --uid joy --display-name "Joy Ning" --subuser joy:swift
{
"user_id": "joy",
"display_name": "Joy Ning",
"email": "",
"suspended": ,
"max_buckets": ,
"auid": ,
"subusers": [
{
"id": "joy:swift",
"permissions": "<none>"
}
],
"keys": [
{
"user": "joy",
"access_key": "4YGLC3480T3Z5ZRY3UHG",
"secret_key": "UuHLN9nlTofwez8Nz0RVJ60Vl6v6zGVJtjLP04pB"
}
],
"swift_keys": [
{
"user": "joy:swift",
"secret_key": "ofbO1nZ1vqUHQwdNuyUd6zLLYhlbiDxMCILfbJoL"
}
],
"caps": [],
"op_mask": "read, write, delete",
"default_placement": "",
"placement_tags": [],
"bucket_quota": {
"enabled": false,
"check_on_raw": false,
"max_size": -,
"max_size_kb": ,
"max_objects": -
},
"user_quota": {
"enabled": false,
"check_on_raw": false,
"max_size": -,
"max_size_kb": ,
"max_objects": -
},
"temp_url_keys": [],
"type": "rgw"
}
[root@ceph5 ~]# radosgw-admin user create --uid joy --display-name "Joy Ning"
[root@ceph5 ~]# radosgw-admin subuser create --uid joy --display-name "Joy Ning" --subuser joy:swift
三、测试
3.1 修改为full
[root@ceph5 ~]# radosgw-admin subuser modify --uid joy --display-name "Joy Ning" --subuser joy:swift --access full
{
"user_id": "joy",
"display_name": "Joy Ning",
"email": "",
"suspended": ,
"max_buckets": ,
"auid": ,
"subusers": [
{
"id": "joy:swift",
"permissions": "full-control"
}
],
"keys": [
{
"user": "joy",
"access_key": "4YGLC3480T3Z5ZRY3UHG",
"secret_key": "UuHLN9nlTofwez8Nz0RVJ60Vl6v6zGVJtjLP04pB"
}
],
"swift_keys": [
{
"user": "joy:swift",
"secret_key": "ofbO1nZ1vqUHQwdNuyUd6zLLYhlbiDxMCILfbJoL"
}
],
"caps": [],
"op_mask": "read, write, delete",
"default_placement": "",
"placement_tags": [],
"bucket_quota": {
"enabled": false,
"check_on_raw": false,
"max_size": -,
"max_size_kb": ,
"max_objects": -
},
"user_quota": {
"enabled": false,
"check_on_raw": false,
"max_size": -,
"max_size_kb": ,
"max_objects": -
},
"temp_url_keys": [],
"type": "rgw"
}
3.2 客户端测试
[root@ceph1 ~]# swift -A http://ceph5.lab.example.com/auth/v1.0 -U joy:swift -K ofbO1nZ1vqUHQwdNuyUd6zLLYhlbiDxMCILfbJoL list
[root@ceph1 ~]# swift -A http://ceph5.lab.example.com/auth/v1.0 -U joy:swift -K ofbO1nZ1vqUHQwdNuyUd6zLLYhlbiDxMCILfbJoL post testswift
[root@ceph1 ~]# swift -A http://ceph5.lab.example.com/auth/v1.0 -U joy:swift -K ofbO1nZ1vqUHQwdNuyUd6zLLYhlbiDxMCILfbJoL upload testswift /etc/ceph/ceph.client.rbd.keyring
etc/ceph/ceph.client.rbd.keyring
3.3 服务端
出现元数据和数据的池
[root@ceph5 ~]# ceph osd pool ls
[root@ceph5 ~]# rados -p fancheng.rgw.buckets.data ls --cluster backup
四 、ceph2配置secondary域
4.1 拉取realm
[root@ceph2 ~]# radosgw-admin realm pull --url http://ceph5.lab.example.com --access-key 4YGLC3480T3Z5ZRY3UHG --secret UuHLN9nlTofwez8Nz0RVJ60Vl6v6zGVJtjLP04pB
-- ::03.055756 7f1974483c40 error read_lastest_epoch .rgw.root:periods.1c6ccdef-ca02-44b5-b212-ba68acbd6aad.latest_epoch
-- ::03.075671 7f1974483c40 Set the period's master zonegroup 5acbb712-0f7f-4108-8d93-ea75a19e33b3 as the default
{
"id": "d4668fc2-ceed-4eb2-a5e7-a70c2aa7deba",
"name": "hubei",
"current_period": "1c6ccdef-ca02-44b5-b212-ba68acbd6aad",
"epoch":
}
[root@ceph2 ~]# radosgw-admin realm list
[root@ceph2 ~]# radosgw-admin zonegroup get --rgw-realm hubei
{
"id": "5acbb712-0f7f-4108-8d93-ea75a19e33b3",
"name": "xiangyang",
"api_name": "xiangyang",
"is_master": "true",
"endpoints": [],
"hostnames": [],
"hostnames_s3website": [],
"master_zone": "b337a562-5921-46fc-aad2-e70e99454e5f",
"zones": [
{
"id": "b337a562-5921-46fc-aad2-e70e99454e5f",
"name": "fancheng",
"endpoints": [
"http://ceph5.lab.example.com"
],
"log_meta": "false",
"log_data": "false",
"bucket_index_max_shards": ,
"read_only": "false",
"tier_type": "",
"sync_from_all": "true",
"sync_from": []
}
],
"placement_targets": [
{
"name": "default-placement",
"tags": []
}
],
"default_placement": "default-placement",
"realm_id": "d4668fc2-ceed-4eb2-a5e7-a70c2aa7deba"
}
4.2 拉取period
[root@ceph2 ~]# radosgw-admin period pull --url http://ceph5.lab.example.com --access-key 4YGLC3480T3Z5ZRY3UHG --secret UuHLN9nlTofwez8Nz0RVJ60Vl6v6zGVJtjLP04pB
-- ::05.406374 7f3dc85c7c40 found existing latest_epoch >= given epoch , returning r=-
{
"id": "1c6ccdef-ca02-44b5-b212-ba68acbd6aad",
"epoch": ,
"predecessor_uuid": "55fafff6-9c4c-4d54-8801-ad48861221a1",
"sync_status": [],
"period_map": {
"id": "1c6ccdef-ca02-44b5-b212-ba68acbd6aad",
"zonegroups": [
{
"id": "5acbb712-0f7f-4108-8d93-ea75a19e33b3",
"name": "xiangyang",
"api_name": "xiangyang",
"is_master": "true",
"endpoints": [],
"hostnames": [],
"hostnames_s3website": [],
"master_zone": "b337a562-5921-46fc-aad2-e70e99454e5f",
"zones": [
{
"id": "b337a562-5921-46fc-aad2-e70e99454e5f",
"name": "fancheng",
"endpoints": [
"http://ceph5.lab.example.com"
],
"log_meta": "false",
"log_data": "false",
"bucket_index_max_shards": ,
"read_only": "false",
"tier_type": "",
"sync_from_all": "true",
"sync_from": []
}
],
"placement_targets": [
{
"name": "default-placement",
"tags": []
}
],
"default_placement": "default-placement",
"realm_id": "d4668fc2-ceed-4eb2-a5e7-a70c2aa7deba"
}
],
"short_zone_ids": [
{
"key": "b337a562-5921-46fc-aad2-e70e99454e5f",
"val":
}
]
},
"master_zonegroup": "5acbb712-0f7f-4108-8d93-ea75a19e33b3",
"master_zone": "b337a562-5921-46fc-aad2-e70e99454e5f",
"period_config": {
"bucket_quota": {
"enabled": false,
"check_on_raw": false,
"max_size": -,
"max_size_kb": ,
"max_objects": -
},
"user_quota": {
"enabled": false,
"check_on_raw": false,
"max_size": -,
"max_size_kb": ,
"max_objects": -
}
},
"realm_id": "d4668fc2-ceed-4eb2-a5e7-a70c2aa7deba",
"realm_name": "hubei",
"realm_epoch":
}
period
4.3 创建Secondary Zone
[root@ceph2 ~]# radosgw-admin zone create --rgw-zonegroup xiangyang --rgw-zone xiantao --endpoints http://ceph2.lab.example.com --default --access-key 4YGLC3480T3Z5ZRY3UHG --secret UuHLN9nlTofwez8Nz0RVJ60Vl6v6zGVJtjLP04pB
-- ::47.924379 7f7d2758cc40 failed reading obj info from .rgw.root:zone_info.b337a562--46fc-aad2-e70e99454e5f: () No such file or directory
-- ::47.924419 7f7d2758cc40 WARNING: could not read zone params for zone id=b337a562--46fc-aad2-e70e99454e5f name=fancheng
{
"id": "b012d15d-a83c-4553-9ec3-09bf45d4a67b",
"name": "xiantao",
"domain_root": "xiantao.rgw.meta:root",
"control_pool": "xiantao.rgw.control",
"gc_pool": "xiantao.rgw.log:gc",
"lc_pool": "xiantao.rgw.log:lc",
"log_pool": "xiantao.rgw.log",
"intent_log_pool": "xiantao.rgw.log:intent",
"usage_log_pool": "xiantao.rgw.log:usage",
"reshard_pool": "xiantao.rgw.log:reshard",
"user_keys_pool": "xiantao.rgw.meta:users.keys",
"user_email_pool": "xiantao.rgw.meta:users.email",
"user_swift_pool": "xiantao.rgw.meta:users.swift",
"user_uid_pool": "xiantao.rgw.meta:users.uid",
"system_key": {
"access_key": "4YGLC3480T3Z5ZRY3UHG",
"secret_key": "UuHLN9nlTofwez8Nz0RVJ60Vl6v6zGVJtjLP04pB"
},
"placement_pools": [
{
"key": "default-placement",
"val": {
"index_pool": "xiantao.rgw.buckets.index",
"data_pool": "xiantao.rgw.buckets.data",
"data_extra_pool": "xiantao.rgw.buckets.non-ec",
"index_type": ,
"compression": ""
}
}
],
"metadata_heap": "",
"tier_config": [],
"realm_id": "d4668fc2-ceed-4eb2-a5e7-a70c2aa7deba"
}
zone-new
[root@ceph2 ~]# radosgw-admin zonegroup get --rgw-realm hubei --rgw-zonegroup
parse error setting 'rgw_zonegroup' to '' (Option --rgw-zonegroup requires an argument.
) {
"id": "5acbb712-0f7f-4108-8d93-ea75a19e33b3",
"name": "xiangyang",
"api_name": "xiangyang",
"is_master": "true",
"endpoints": [],
"hostnames": [],
"hostnames_s3website": [],
"master_zone": "b337a562-5921-46fc-aad2-e70e99454e5f",
"zones": [
{
"id": "b012d15d-a83c-4553-9ec3-09bf45d4a67b",
"name": "xiantao",
"endpoints": [
"http://ceph2.lab.example.com"
],
"log_meta": "false",
"log_data": "true",
"bucket_index_max_shards": ,
"read_only": "false",
"tier_type": "",
"sync_from_all": "true",
"sync_from": []
},
{
"id": "b337a562-5921-46fc-aad2-e70e99454e5f",
"name": "fancheng",
"endpoints": [
"http://ceph5.lab.example.com"
],
"log_meta": "false",
"log_data": "true",
"bucket_index_max_shards": ,
"read_only": "false",
"tier_type": "",
"sync_from_all": "true",
"sync_from": []
}
],
"placement_targets": [
{
"name": "default-placement",
"tags": []
}
],
"default_placement": "default-placement",
"realm_id": "d4668fc2-ceed-4eb2-a5e7-a70c2aa7deba"
}
[root@ceph5 ~]# radosgw-admin zone modify --rgw-zonegroup xiangyang --rgw-zone fancheng --endpoints http://ceph5.lab.example.com
{
"id": "b337a562-5921-46fc-aad2-e70e99454e5f",
"name": "fancheng",
"domain_root": "fancheng.rgw.meta:root",
"control_pool": "fancheng.rgw.control",
"gc_pool": "fancheng.rgw.log:gc",
"lc_pool": "fancheng.rgw.log:lc",
"log_pool": "fancheng.rgw.log",
"intent_log_pool": "fancheng.rgw.log:intent",
"usage_log_pool": "fancheng.rgw.log:usage",
"reshard_pool": "fancheng.rgw.log:reshard",
"user_keys_pool": "fancheng.rgw.meta:users.keys",
"user_email_pool": "fancheng.rgw.meta:users.email",
"user_swift_pool": "fancheng.rgw.meta:users.swift",
"user_uid_pool": "fancheng.rgw.meta:users.uid",
"system_key": {
"access_key": "",
"secret_key": "BiGRHcqL03RFoFWAbymFZj6xDQAXYFVb60cIzvav"
},
"placement_pools": [
{
"key": "default-placement",
"val": {
"index_pool": "fancheng.rgw.buckets.index",
"data_pool": "fancheng.rgw.buckets.data",
"data_extra_pool": "fancheng.rgw.buckets.non-ec",
"index_type": ,
"compression": ""
}
}
],
"metadata_heap": "",
"tier_config": [],
"realm_id": "d4668fc2-ceed-4eb2-a5e7-a70c2aa7deba"
}
4.4 更新period
[root@ceph5 ~]# radosgw-admin period update --commit
{
"id": "1c6ccdef-ca02-44b5-b212-ba68acbd6aad",
"epoch": ,
"predecessor_uuid": "55fafff6-9c4c-4d54-8801-ad48861221a1",
"sync_status": [],
"period_map": {
"id": "1c6ccdef-ca02-44b5-b212-ba68acbd6aad",
"zonegroups": [
{
"id": "5acbb712-0f7f-4108-8d93-ea75a19e33b3",
"name": "xiangyang",
"api_name": "xiangyang",
"is_master": "true",
"endpoints": [],
"hostnames": [],
"hostnames_s3website": [],
"master_zone": "b337a562-5921-46fc-aad2-e70e99454e5f",
"zones": [
{
"id": "b337a562-5921-46fc-aad2-e70e99454e5f",
"name": "fancheng",
"endpoints": [
"http://ceph5.lab.example.com"
],
"log_meta": "false",
"log_data": "false",
"bucket_index_max_shards": ,
"read_only": "false",
"tier_type": "",
"sync_from_all": "true",
"sync_from": []
}
],
"placement_targets": [
{
"name": "default-placement",
"tags": []
}
],
"default_placement": "default-placement",
"realm_id": "d4668fc2-ceed-4eb2-a5e7-a70c2aa7deba"
}
],
"short_zone_ids": [
{
"key": "b337a562-5921-46fc-aad2-e70e99454e5f",
"val":
}
]
},
"master_zonegroup": "5acbb712-0f7f-4108-8d93-ea75a19e33b3",
"master_zone": "b337a562-5921-46fc-aad2-e70e99454e5f",
"period_config": {
"bucket_quota": {
"enabled": false,
"check_on_raw": false,
"max_size": -,
"max_size_kb": ,
"max_objects": -
},
"user_quota": {
"enabled": false,
"check_on_raw": false,
"max_size": -,
"max_size_kb": ,
"max_objects": -
}
},
"realm_id": "d4668fc2-ceed-4eb2-a5e7-a70c2aa7deba",
"realm_name": "hubei",
"realm_epoch":
}
[root@ceph2 ~]# ceph auth get-or-create client.rgw.ceph2 mon 'allow rwx' osd 'allow rwx' -o /etc/ceph/ceph.client.radosgw.keyring --cluster ceph
[root@ceph2 ~]# ceph osd pool ls
发现没有成功，重新拉取一遍
4.5 删除relam
[root@ceph2 ~]# radosgw-admin zonegroup delete --rgw-zonegroup xiangyang
[root@ceph2 ~]# radosgw-admin realm delete --rgw-realm hubei
[root@ceph2 ~]# radosgw-admin zone delete --rgw-zone xiantao
[root@ceph2 ~]# radosgw-admin realm list
4.6 重新拉取
[root@ceph2 ~]# radosgw-admin realm pull --url http://ceph5.lab.example.com --access-key 4YGLC3480T3Z5ZRY3UHG --secret UuHLN9nlTofwez8Nz0RVJ60Vl6v6zGVJtjLP04pB
-- ::24.324288 7f00bbc65c40 Set the period's master zonegroup 5acbb712-0f7f-4108-8d93-ea75a19e33b3 as the default
{
"id": "d4668fc2-ceed-4eb2-a5e7-a70c2aa7deba",
"name": "hubei",
"current_period": "1c6ccdef-ca02-44b5-b212-ba68acbd6aad",
"epoch":
} [root@ceph2 ~]# radosgw-admin period pull --url http://ceph5.lab.example.com --access-key 4YGLC3480T3Z5ZRY3UHG --secret UuHLN9nlTofwez8Nz0RVJ60Vl6v6zGVJtjLP04pB
{
"id": "1c6ccdef-ca02-44b5-b212-ba68acbd6aad",
"epoch": ,
"predecessor_uuid": "55fafff6-9c4c-4d54-8801-ad48861221a1",
"sync_status": [],
"period_map": {
"id": "1c6ccdef-ca02-44b5-b212-ba68acbd6aad",
"zonegroups": [
{
"id": "5acbb712-0f7f-4108-8d93-ea75a19e33b3",
"name": "xiangyang",
"api_name": "xiangyang",
"is_master": "true",
"endpoints": [],
"hostnames": [],
"hostnames_s3website": [],
"master_zone": "b337a562-5921-46fc-aad2-e70e99454e5f",
"zones": [
{
"id": "b337a562-5921-46fc-aad2-e70e99454e5f",
"name": "fancheng",
"endpoints": [
"http://ceph5.lab.example.com"
],
"log_meta": "false",
"log_data": "false",
"bucket_index_max_shards": ,
"read_only": "false",
"tier_type": "",
"sync_from_all": "true",
"sync_from": []
}
],
"placement_targets": [
{
"name": "default-placement",
"tags": []
}
],
"default_placement": "default-placement",
"realm_id": "d4668fc2-ceed-4eb2-a5e7-a70c2aa7deba"
}
],
"short_zone_ids": [
{
"key": "b337a562-5921-46fc-aad2-e70e99454e5f",
"val":
}
]
},
"master_zonegroup": "5acbb712-0f7f-4108-8d93-ea75a19e33b3",
"master_zone": "b337a562-5921-46fc-aad2-e70e99454e5f",
"period_config": {
"bucket_quota": {
"enabled": false,
"check_on_raw": false,
"max_size": -,
"max_size_kb": ,
"max_objects": -
},
"user_quota": {
"enabled": false,
"check_on_raw": false,
"max_size": -,
"max_size_kb": ,
"max_objects": -
}
},
"realm_id": "d4668fc2-ceed-4eb2-a5e7-a70c2aa7deba",
"realm_name": "hubei",
"realm_epoch":
} -- ::54.816633 7efd7ebf8c40 found existing latest_epoch >= given epoch , returning r=-
[root@ceph2 ~]# radosgw-admin zone create --rgw-zonegroup xiangyang --rgw-zone xiantao --endpoints http://ceph2.lab.example.com --default --access-key 4YGLC3480T3Z5ZRY3UHG --secret UuHLN9nlTofwez8Nz0RVJ60Vl6v6zGVJtjLP04pB
-- ::01.076567 7fb699f77c40 failed reading obj info from .rgw.root:zone_info.b337a562--46fc-aad2-e70e99454e5f: () No such file or directory
-- ::01.076625 7fb699f77c40 WARNING: could not read zone params for zone id=b337a562--46fc-aad2-e70e99454e5f name=fancheng
{
"id": "09ece789-b2fc-4379-8c60-d37811074216",
"name": "xiantao",
"domain_root": "xiantao.rgw.meta:root",
"control_pool": "xiantao.rgw.control",
"gc_pool": "xiantao.rgw.log:gc",
"lc_pool": "xiantao.rgw.log:lc",
"log_pool": "xiantao.rgw.log",
"intent_log_pool": "xiantao.rgw.log:intent",
"usage_log_pool": "xiantao.rgw.log:usage",
"reshard_pool": "xiantao.rgw.log:reshard",
"user_keys_pool": "xiantao.rgw.meta:users.keys",
"user_email_pool": "xiantao.rgw.meta:users.email",
"user_swift_pool": "xiantao.rgw.meta:users.swift",
"user_uid_pool": "xiantao.rgw.meta:users.uid",
"system_key": {
"access_key": "4YGLC3480T3Z5ZRY3UHG",
"secret_key": "UuHLN9nlTofwez8Nz0RVJ60Vl6v6zGVJtjLP04pB"
},
"placement_pools": [
{
"key": "default-placement",
"val": {
"index_pool": "xiantao.rgw.buckets.index",
"data_pool": "xiantao.rgw.buckets.data",
"data_extra_pool": "xiantao.rgw.buckets.non-ec",
"index_type": ,
"compression": ""
}
}
],
"metadata_heap": "",
"tier_config": [],
"realm_id": "d4668fc2-ceed-4eb2-a5e7-a70c2aa7deba"
}
[root@ceph2 ~]# cd /etc/ceph/
[root@ceph2 ceph]# rm -rf ceph.client.radosgw.keyring
[root@ceph2 ceph]# ceph auth get-or-create client.rgw.ceph2 mon 'allow rwx' osd 'allow rwx' -o /etc/ceph/ceph.client.radosgw.keyring --cluster ceph
[root@ceph2 ceph]# ceph osd pool ls
testpool
rbd
rbdmirror
.rgw.root
default.rgw.control
default.rgw.meta
default.rgw.log
xiantao.rgw.control
xiantao.rgw.meta
xiantao.rgw.log
五、迁移Single-Zone至Multi-Zone
5.1 查看
[root@ceph1 ~]# swift -A http://ceph5.lab.example.com/auth/v1.0 -U joy:swift -K ofbO1nZ1vqUHQwdNuyUd6zLLYhlbiDxMCILfbJoL list
[root@ceph1 ~]# swift -A http://ceph5.lab.example.com/auth/v1.0 -U joy:swift -K ofbO1nZ1vqUHQwdNuyUd6zLLYhlbiDxMCILfbJoL list testswift
5.2 ceph5把default的zone添加到一个realm
[root@ceph5 ~]# radosgw-admin realm list
[root@ceph5 ~]# radosgw-admin zonegroup list
5.3 为default创建一个新的zonegroup
[root@ceph5 ~]# radosgw-admin realm create --rgw-realm realmnew
5.4 修改default的realm，设置为master
[root@ceph5 ~]# radosgw-admin zonegroup modify --rgw-realm realmnew --rgw-zonegroup default --master
{
"id": "e80133e1-a513-44f5-ba90-e25b6c987b26",
"name": "default",
"api_name": "",
"is_master": "true",
"endpoints": [],
"hostnames": [],
"hostnames_s3website": [],
"master_zone": "1b85c5b1-19d2-48a1-bb45-3ac75895aeed",
"zones": [
{
"id": "1b85c5b1-19d2-48a1-bb45-3ac75895aeed",
"name": "default",
"endpoints": [],
"log_meta": "false",
"log_data": "false",
"bucket_index_max_shards": ,
"read_only": "false",
"tier_type": "",
"sync_from_all": "true",
"sync_from": []
}
],
"placement_targets": [
{
"name": "default-placement",
"tags": []
}
],
"default_placement": "default-placement",
"realm_id": "de918d45-d763-416d-af0a-0350b1339ca1"
}
[root@ceph5 ~]# radosgw-admin zone modify --rgw-zonegroup default --rgw-zone default --master --endpoints http://ceph5.lab.example.com
{
"id": "1b85c5b1-19d2-48a1-bb45-3ac75895aeed",
"name": "default",
"domain_root": "default.rgw.meta:root",
"control_pool": "default.rgw.control",
"gc_pool": "default.rgw.log:gc",
"lc_pool": "default.rgw.log:lc",
"log_pool": "default.rgw.log",
"intent_log_pool": "default.rgw.log:intent",
"usage_log_pool": "default.rgw.log:usage",
"reshard_pool": "default.rgw.log:reshard",
"user_keys_pool": "default.rgw.meta:users.keys",
"user_email_pool": "default.rgw.meta:users.email",
"user_swift_pool": "default.rgw.meta:users.swift",
"user_uid_pool": "default.rgw.meta:users.uid",
"system_key": {
"access_key": "",
"secret_key": ""
},
"placement_pools": [
{
"key": "default-placement",
"val": {
"index_pool": "default.rgw.buckets.index",
"data_pool": "default.rgw.buckets.data",
"data_extra_pool": "default.rgw.buckets.non-ec",
"index_type": ,
"compression": ""
}
}
],
"metadata_heap": "",
"tier_config": [],
"realm_id": ""
}
5.5 创建一个系统用户
[root@ceph5 ~]# radosgw-admin user create --uid syncuser1 --display-name "sync user" --rgw-zonegroup default --rgw-zone default
{
"user_id": "syncuser1",
"display_name": "sync user",
"email": "",
"suspended": ,
"max_buckets": ,
"auid": ,
"subusers": [],
"keys": [
{
"user": "syncuser1",
"access_key": "MIVNCAI762F49VPRAFDF",
"secret_key": "kHJSrkBxpVQJsCOy2sV4P9ElmzTkjcPX81R6hycR"
}
],
"swift_keys": [],
"caps": [],
"op_mask": "read, write, delete",
"default_placement": "",
"placement_tags": [],
"bucket_quota": {
"enabled": false,
"check_on_raw": false,
"max_size": -,
"max_size_kb": ,
"max_objects": -
},
"user_quota": {
"enabled": false,
"check_on_raw": false,
"max_size": -,
"max_size_kb": ,
"max_objects": -
},
"temp_url_keys": [],
"type": "rgw"
}
5.6 将master-zone与系统用户关联
[root@ceph5 ~]# radosgw-admin zone modify --rgw-zonegroup default --rgw-zone default --master --endpoints http://ceph5.lab.example.com --access-key MIVNCAI762F49VPRAFDF --secret kHJSrkBxpVQJsCOy2sV4P9ElmzTkjcPX81R6hycR
{
"id": "1b85c5b1-19d2-48a1-bb45-3ac75895aeed",
"name": "default",
"domain_root": "default.rgw.meta:root",
"control_pool": "default.rgw.control",
"gc_pool": "default.rgw.log:gc",
"lc_pool": "default.rgw.log:lc",
"log_pool": "default.rgw.log",
"intent_log_pool": "default.rgw.log:intent",
"usage_log_pool": "default.rgw.log:usage",
"reshard_pool": "default.rgw.log:reshard",
"user_keys_pool": "default.rgw.meta:users.keys",
"user_email_pool": "default.rgw.meta:users.email",
"user_swift_pool": "default.rgw.meta:users.swift",
"user_uid_pool": "default.rgw.meta:users.uid",
"system_key": {
"access_key": "MIVNCAI762F49VPRAFDF",
"secret_key": "kHJSrkBxpVQJsCOy2sV4P9ElmzTkjcPX81R6hycR"
},
"placement_pools": [
{
"key": "default-placement",
"val": {
"index_pool": "default.rgw.buckets.index",
"data_pool": "default.rgw.buckets.data",
"data_extra_pool": "default.rgw.buckets.non-ec",
"index_type": ,
"compression": ""
}
}
],
"metadata_heap": "",
"tier_config": [],
"realm_id": ""
}
5.7 提交新的配置
[root@ceph5 ~]# radosgw-admin period update --commit --rgw-zonegroup default --rgw-zone default --rgw-realm realmnew
-- ::39.137059 7fd1dd9cec40 Set the period's master zonegroup e80133e1-a513-44f5-ba90-e25b6c987b26 as the default
{
"id": "152a15c4-d0f7-4bd1-93db-fc1f8655b741",
"epoch": ,
"predecessor_uuid": "0daf85ed-732b-4f0f-8f58-a3f8ef1e996e",
"sync_status": [],
"period_map": {
"id": "152a15c4-d0f7-4bd1-93db-fc1f8655b741",
"zonegroups": [
{
"id": "e80133e1-a513-44f5-ba90-e25b6c987b26",
"name": "default",
"api_name": "",
"is_master": "true",
"endpoints": [],
"hostnames": [],
"hostnames_s3website": [],
"master_zone": "1b85c5b1-19d2-48a1-bb45-3ac75895aeed",
"zones": [
{
"id": "1b85c5b1-19d2-48a1-bb45-3ac75895aeed",
"name": "default",
"endpoints": [
"http://ceph5.lab.example.com"
],
"log_meta": "false",
"log_data": "false",
"bucket_index_max_shards": ,
"read_only": "false",
"tier_type": "",
"sync_from_all": "true",
"sync_from": []
}
],
"placement_targets": [
{
"name": "default-placement",
"tags": []
}
],
"default_placement": "default-placement",
"realm_id": "de918d45-d763-416d-af0a-0350b1339ca1"
}
],
"short_zone_ids": [
{
"key": "1b85c5b1-19d2-48a1-bb45-3ac75895aeed",
"val":
}
]
},
"master_zonegroup": "e80133e1-a513-44f5-ba90-e25b6c987b26",
"master_zone": "1b85c5b1-19d2-48a1-bb45-3ac75895aeed",
"period_config": {
"bucket_quota": {
"enabled": false,
"check_on_raw": false,
"max_size": -,
"max_size_kb": ,
"max_objects": -
},
"user_quota": {
"enabled": false,
"check_on_raw": false,
"max_size": -,
"max_size_kb": ,
"max_objects": -
}
},
"realm_id": "de918d45-d763-416d-af0a-0350b1339ca1",
"realm_name": "realmnew",
"realm_epoch":
}
5.8 修改配置文件使生效
[root@ceph5 ~]# vim /etc/ceph/backup.conf
fsid = 51dda18c--4edb-8ba9-27330ead81a7
mon_initial_members = ceph5
mon_host = 172.25.250.14
auth_cluster_required = cephx
auth_service_required = cephx
auth_client_required = cephx
public_network = 172.25.250.0/
cluster_network = 172.25.250.0/
[mgr]
mgr modules = dashboard
[client.rgw.ceph5]
host = ceph5
keyring = /etc/ceph/backup.client.rgw.ceph5.keyring
rgw_frontends = civetweb port= num_threads=
log = /var/log/ceph/$cluster.$name.log
rgw_dns_name = ceph5.lab.example.com
rgw_zone = default
rgw_region = realmnew
rgw_zonegroup = default
[root@ceph5 ~]# systemctl restart ceph-radosgw@rgw.ceph5
[root@ceph5 ~]# ps -ef|grep rados
博主声明：本文的内容来源主要来自誉天教育晏威老师，由本人实验完成操作验证，需要的博友请联系誉天教育（http://www.yutianedu.com/），获得官方同意或者晏老师（https://www.cnblogs.com/breezey/）本人同意即可转载，谢谢！
012 Ceph多区域网关的更多相关文章
- ceph 008 ceph多区域网关(ceph对象容灾) cephfs文件系统
clienta作为集群的管理人员.一部分.他是需要秘钥与配置文件的 但真正服务端只需要通过curl就好 ceph 多区域网关 对象存储容灾解决方案 zone与zone会做数据同步. 把会做同步的rgw ...
- Ceph对象存储网关中的索引工作原理<转>
Ceph 对象存储网关允许你通过 Swift 及 S3 API 访问 Ceph .它将这些 API 请求转化为 librados 请求.Librados 是一个非常出色的对象存储(库)但是它无法高效的 ...
- ceph安装对象网关
1.概述 安装3个网关节点分别是:controller-03.controller-04和controller-05,使用ceph gw自带的Civetweb提供服务,前端使用nginx作为前端代理. ...
- Ceph集群rbd-mirror A、B区域备份实施方案
Ceph集群rbd-mirror A.B区域备份实施方案 备注:首先准备两个集群, 并确认其状态,集群的准备过程在这就不做陈述 1.查看集群状态 A区域 [root@ceph2111 ceph]# c ...
- iSCSI 网关管理 - Storage6
iSCSI网关集成了Ceph存储和iSCSI标准,以提供一个高可用性(HA) iSCSI目标,该目标将RADOS块设备(RBD)映像导出为SCSI磁盘.iSCSI协议允许客户机 (initiator) ...
- Ceph块设备
Ceph块设备 来自这里. 块是一个字节序列(例如,一个512字节的数据块).基于块的存储接口是最常见的存储数据的方法,它通常基于旋转介质,像硬盘.CD.软盘,甚至传统的9道磁带. 基本的块设备命令 ...
- Ceph分布式存储详述
存储发展史 企业中使用存储按照其功能,使用场景,一直在持续发展和迭代,大体上可以分为四个阶段: DAS:Direct Attached Storage,即直连存储,第一代存储系统,通过SCSI总线扩展 ...
- Ceph的Block分析
一个块是一个连续的字节序列(例如一个512字节的连续数据是一个块).基于块的存储接口通常是旋转介质,例如磁盘.光盘.软盘等.块设备接口的普及使得可以用虚拟的块设备成为和大容量数据存储系统交互的接口,如 ...
- 为cloudstack搭建ceph文件系统
1. 安装dell服务器, raid卡需要采用直通模式(non-raid); 各磁盘独立运行. 网络依赖硬件不同,使用万兆网卡或者两个千兆网卡做bonding6. 2. 配置host map(1 ...
随机推荐
- oracle HEXTORAW(c1)
[功能]将一个十六进制构成的字符串转换为二进制 [参数]c1,十六进制的字符串 [返回]字符串 [示例] select HEXTORAW('A123') from dual;
- Java练习 SDUT-1133_模拟计算器
C/C++经典程序训练3---模拟计算器 Time Limit: 1000 ms Memory Limit: 8192 KiB Problem Description 简单计算器模拟:输入两个整数和一 ...
- @总结 - 2@ 位运算卷积/子集卷积 —— FWT/FMT
目录 @0 - 参考资料@ @1 - 异或卷积概念及性质@ @2 - 快速沃尔什正变换(异或)@ @3 - 快速沃尔什逆变换(异或)@ @4 - 与卷积.或卷积@ @5 - 参考代码实现@ @6 - ...
- 1x1卷积
你可能会想为什么有人会用1x1卷积,因为它关注的不是一块像素,而是一个像素,图1 图1 我们看看传统的卷积,它基本上是运行在一个小块图像上的小分类器,但仅仅是个线性分类器.图2 图2 如果你在中间加一 ...
- Linux中使用gcc编译文件
一个项目中可能有多个cpp文件,在linux下编译执行过程如下: g++ main.cpp distance.cpp ./a.out 即可一起编译两个文件,然后执行该程序.
- HZOI20190714 T1序列
什么沙雕题啊……考察的是啥啊,分类咋搞啊……愁死我了…… 先把作者的正解放出来: 序列因为选出的一段是一个等比序列的子序列,我们分为两种情况:1. q=1,相当于找一个最长每个数都相等的子串,这个扫一 ...
- 模板—tarjan缩点
void tarjan(int x) { dfn[x]=++cnt;low[x]=cnt; vi[x]=; stack[++top]=x; for(rint i=f(x);i;i=n(i)) if(! ...
- jq杂项方法/工具方法----trim() html() val() text() attr()
https://www.cnblogs.com/sandraryan/ $.trim() 函数用于去除字符串两端的空白字符.在中间的时候不会去掉. var str = ' 去除字符串左右两端的空格,换 ...
- CSS中常用的简写模式
一.font属性简写 font-style:字体样式 normal 默认值.浏览器显示一个标准的字体样式. italic 浏览器会显示一个斜体的字体样式. oblique 浏览器会显示一个倾斜的字体样 ...
- uni-app 快速认识
uni-app 是一个使用 Vue.js 开发所有前端应用的框架,开发者编写一套代码,可发布到iOS.Android.H5.以及各种小程序(微信/阿里/百度/头条/QQ)等多个平台. 即使不跨端,un ...