010 Ceph RGW对象存储

一、对象存储

1.1 介绍

通过对象存储，将数据存储为对象，每个对象除了包含数据，还包含数据自身的元数据

对象通过Object ID来检索，无法通过普通文件系统操作来直接访问对象，只能通过API来访问，或者第三方客户端（实际上也是对API的封装）

对象存储中的对象不整理到目录树中，而是存储在扁平的命名空间中，Amazon S3将这个扁平命名空间称为bucket。而swift则将其称为容器

无论是bucket还是容器，都不能嵌套

bucket需要被授权才能访问到，一个帐户可以对多个bucket授权，而权限可以不同

对象存储的优点：易扩展、快速检索

1.2 Rados网关介绍

RADOS网关也称为Ceph对象网关、RADOSGW、RGW，是一种服务，使客户端能够利用标准对象存储API来访问Ceph集群。它支持S3和Swift API

rgw运行于librados之上，事实上就是一个称之为Civetweb的web服务器来响应api请求

客户端使用标准api与rgw通信，而rgw则使用librados与ceph集群通信

rgw客户端通过s3或者swift api使用rgw用户进行身份验证。然后rgw网关代表用户利用cephx与ceph存储进行身份验证

二、RADOS网关部署

2.1 配置radosgw

[root@ceph5 ~]#  ceph auth get-or-create client.rgw.ceph5  mon 'allow rwx' osd 'allow rwx' -o /etc/ceph/backup.client.rgw.ceph5.keyring --cluster backup

[root@ceph5 ~]# vim /etc/ceph/backup.conf

fsid = 51dda18c--4edb-8ba9-27330ead81a7
mon_initial_members = ceph5
mon_host = 172.25.250.14

auth_cluster_required = cephx
auth_service_required = cephx
auth_client_required = cephx

public_network = 172.25.250.0/
cluster_network = 172.25.250.0/

[mgr]
mgr modules = dashboard

[client.rgw.ceph5]
host = ceph5
keyring = /etc/ceph/backup.client.rgw.ceph5.keyring
rgw_frontends = civetweb port=

[root@ceph5 ~]# systemctl restart ceph-radosgw@rgw.ceph5

[root@ceph5 ~]# ps -ef|grep rados

root             : ?        :: /usr/bin/radosgw -f --cluster backup --name client.rgw.ceph5 --setuser ceph --setgroup ceph

[root@ceph5 ~]# netstat -ntlp|grep 80

tcp               0.0.0.0:              0.0.0.0:*               LISTEN      /radosgw

[root@ceph5 ~]# ceph osd pool ls

[root@ceph5 ~]# ceph -s

[root@ceph5 ~]# ceph osd pool application enable rbd rbd

[root@ceph5 ~]# ceph osd pool application enable rbdmirror rbd

[root@ceph5 ~]# ceph -s

[root@ceph5 ~]#  cat /usr/lib/systemd/system/ceph-radosgw@.service

[Unit]
Description=Ceph rados gateway
After=network-online.target local-fs.target time-sync.target
Wants=network-online.target local-fs.target time-sync.target
PartOf=ceph-radosgw.target

[Service]
LimitNOFILE=
LimitNPROC=
EnvironmentFile=-/etc/sysconfig/ceph
Environment=CLUSTER=ceph
ExecStart=/usr/bin/radosgw -f --cluster ${CLUSTER} --name client.%i --setuser ceph --setgroup ceph
PrivateDevices=yes
ProtectHome=true
ProtectSystem=full
PrivateTmp=true
TasksMax=infinity
Restart=on-failure
StartLimitInterval=30s
StartLimitBurst=

[Install]
WantedBy=ceph-radosgw.target

2.2 定义监听的线程数

[root@ceph5 ~]# vim /etc/ceph/backup.conf

[client.rgw.ceph5]
host = ceph5
keyring = /etc/ceph/backup.client.rgw.ceph5.keyring
rgw_frontends = civetweb port= num_threads=
log = /var/log/ceph/$cluster.$name.log

[root@ceph5 ~]# systemctl restart ceph-radosgw@rgw.ceph5

[root@ceph5 ~]# ps -ef|grep rados

ceph    : ? :: /usr/bin/radosgw -f --cluster backup --name client.rgw.ceph5 --setuser ceph --setgroup ceph

2.3 访问rados网关

[root@ceph5 ~]# curl http://ceph5

<?xml version="1.0" encoding="UTF-8"?><ListAllMyBucketsResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Owner><ID>anonymous</ID><DisplayName></DisplayName></Owner><Buckets></Buckets></ListAllMyBucketsResult>

三、S3对象存储

3.1 S3简介

S3由Amazon于2006年推出，全称为Simple Storage Service

S3定义了对象存储，是对象存储事实上的标准，从某种意义上说，S3就是对象存储，对象存储就是S3

S3是对象存储市场的霸主，后续的对象存储都是对S3的模仿

3.2 用户以及权限设置

创建radosgw的用户

[root@ceph5 ~]#  radosgw-admin user create --uid joy --display-name 'Joy Ning'

{
    "user_id": "joy",
    "display_name": "Joy Ning",
    "email": "",
    "suspended": ,
    "max_buckets": ,
    "auid": ,
    "subusers": [],
    "keys": [
        {
            "user": "joy",
            "access_key": "X0CVIF04TAJVTN9D29UL",
            "secret_key": "vMmPqPap0FC0IRC5J3t9AIPgXNoiw1H9TOWELd5B"
        }
    ],
    "swift_keys": [],
    "caps": [],
    "op_mask": "read, write, delete",
    "default_placement": "",
    "placement_tags": [],
    "bucket_quota": {
        "enabled": false,
        "check_on_raw": false,
        "max_size": -,
        "max_size_kb": ,
        "max_objects": -
    },
    "user_quota": {
        "enabled": false,
        "check_on_raw": false,
        "max_size": -,
        "max_size_kb": ,
        "max_objects": -
    },
    "temp_url_keys": [],
    "type": "rgw"
}

"Joy Ning"

修改信息

[root@ceph5 ~]# radosgw-admin user modify --uid joy --display-name 'joy Ningrui'  --max_buckets 2000

禁用suspend

[root@ceph5 ~]# radosgw-admin user suspend --uid joy

启用

[root@ceph5 ~]# radosgw-admin user enable --uid joy

列出用户

[root@ceph5 ~]# radosgw-admin user list

删除用户

[root@ceph5 ~]# radosgw-admin user rm --uid joy

[root@ceph5 ~]# radosgw-admin user list

[root@ceph5 ~]#  radosgw-admin user create --uid joy --display-name 'Joy Ning'

{
    "user_id": "joy",
    "display_name": "Joy Ning",
    "email": "",
    "suspended": ,
    "max_buckets": ,
    "auid": ,
    "subusers": [],
    "keys": [
        {
            "user": "joy",
            "access_key": "5XCV68WUQJFFJPVM3UHK",
            "secret_key": "xhaA2YB1CA3xH54xLbmwPcglqjDyuFez36F8XGuG"
        }
    ],
    "swift_keys": [],
    "caps": [],
    "op_mask": "read, write, delete",
    "default_placement": "",
    "placement_tags": [],
    "bucket_quota": {
        "enabled": false,
        "check_on_raw": false,
        "max_size": -,
        "max_size_kb": ,
        "max_objects": -
    },
    "user_quota": {
        "enabled": false,
        "check_on_raw": false,
        "max_size": -,
        "max_size_kb": ,
        "max_objects": -
    },
    "temp_url_keys": [],
    "type": "rgw"
}

[root@ceph5 ~]# radosgw-admin key create --uid joy --display-name 'Joy Ning' --key-type=s3  --gen-access-key --gen-secret

删除key

[root@ceph5 ~]# radosgw-admin key rm --uid joy --display-name 'Joy Ning' --key-type=s3 --access-key HPT1SBAXCXW46ZACKPY0

3.3 设置配额

基于用户的配额

[root@ceph5 ~]# radosgw-admin quota set --quota-scope=user --uid=joy --max-size 1

[root@ceph5 ~]# radosgw-admin user info --uid joy

开启配额

[root@ceph5 ~]# radosgw-admin quota enable --quota-scope=user --uid joy

[root@ceph5 ~]# radosgw-admin user info --uid joy

[root@ceph5 ~]# radosgw-admin quota set --quota-scope=bucket --uid=joy --max-size 1

[root@ceph5 ~]# radosgw-admin quota enable --quota-scope=bucket --uid=joy

[root@ceph5 ~]# radosgw-admin user info --uid joy

注：如果两个都进行配置，则那个先到，使用哪一个

关闭配额

可以disable

[root@ceph5 ~]# radosgw-admin quota disable  --quota-scope=bucket --uid=joy

也可以参数设为1

[root@ceph5 ~]# radosgw-admin quota set --quota-scope=user --uid joy --max-size -1

[root@ceph5 ~]# radosgw-admin user info --uid joy

3.4 统计数据

统计所有

[root@ceph5 ~]# radosgw-admin usage show --uid joy

[root@ceph5 ~]# radosgw-admin usage show --uid joy --start-date 2019-03-19 21:00:00 --end-date 2019-03-19 22:00:00

3.5 利用rados网关来访问s3对象

[root@ceph5 ~]#  vim /etc/ceph/backup.conf

[root@ceph5 ~]# systemctl restart ceph-radosgw@rgw.ceph5
[root@ceph5 ~]# ps -ef|grep rados

ceph             : ?        :: /usr/bin/radosgw -f --cluster backup --name client.rgw.ceph5 --setuser ceph --setgroup ceph

四 验证配置

4.1 配置s3cmd

[root@ceph1 ceph]# yum -y install s3cmd

[root@ceph1 ceph]# s3cmd --configure

root@ceph6's password:
Permission denied, please try again.
root@ceph6's password:
hosts                                                                                                                %       .6MB/s   :
[root@ceph1 ceph]# s3cmd --configure

Enter new values or accept defaults in brackets with Enter.
Refer to user manual for detailed description of all options.

Access key and Secret key are your identifiers for Amazon S3. Leave them empty for using the env variables.
Access Key: 5XCV68WUQJFFJPVM3UHK
Secret Key: xhaA2YB1CA3xH54xLbmwPcglqjDyuFez36F8XGuG
Default Region [US]: 

Encryption password is used to protect your files from reading
by unauthorized persons while in transfer to S3
Encryption password: redhat
Path to GPG program [/usr/bin/gpg]: 

When using secure HTTPS protocol all communication with Amazon S3
servers is protected from 3rd party eavesdropping. This method is
slower than plain HTTP, and can only be proxied with Python 2.7 or newer
Use HTTPS protocol [Yes]: no

On some networks all internet access must go through a HTTP proxy.
Try setting it here if you can't connect to S3 directly
HTTP Proxy server name: ceph5.lab.example.com
HTTP Proxy server port []: 

New settings:
  Access Key: 5XCV68WUQJFFJPVM3UHK
  Secret Key: xhaA2YB1CA3xH54xLbmwPcglqjDyuFez36F8XGuG
  Default Region: US
  Encryption password: redhat
  Path to GPG program: /usr/bin/gpg
  Use HTTPS protocol: False
  HTTP Proxy server name: ceph5.lab.example.com
  HTTP Proxy server port: 

Test access with supplied credentials? [Y/n] n

Save settings? [y/N] y
Configuration saved to '/root/.s3cfg'

[root@ceph1 ceph]# vim /root/.s3cfg

host_base = ceph5
host_bucket = %(bucket)s.ceph5.lab.example.com
cloudfront_host = cloudfront.amazonaws.com
website_endpoint = http://%(bucket)s.ceph5.lab.example.com/

4.2 创建bucket

[root@ceph1 ceph]# s3cmd mb s3://test

4.3 传送数据

[root@ceph1 ceph]# echo 11111 >/tmp/demoobject

[root@ceph1 ceph]# s3cmd put --acl-public /tmp/demoobject s3://test/demoobject

[root@ceph1 ceph]# vim /etc/hosts

172.25.250.10  ceph1    ceph1.lab.example.com servera
172.25.250.11  ceph2    ceph2.lab.example.com serverb
172.25.250.12  ceph3    ceph3.lab.example.com serverc
172.25.250.13  ceph4    ceph4.lab.example.com serverd
172.25.250.14  ceph5    ceph5.lab.example.com servere  test.ceph5.lab.example.com

4.4 访问bucket

[root@ceph1 ceph]# curl http://test.ceph5.lab.example.com/demoobject

4.5 查看bucket

到服务端看

[root@ceph5 ~]# radosgw-admin bucket list

[root@ceph5 ~]# radosgw-admin bucket stats --bucket=test

{
    "bucket": "test",
    "zonegroup": "e80133e1-a513-44f5-ba90-e25b6c987b26",
    "placement_rule": "default-placement",
    "explicit_placement": {
        "data_pool": "",
        "data_extra_pool": "",
        "index_pool": ""
    },
    "id": "1b85c5b1-19d2-48a1-bb45-3ac75895aeed.4235.1",
    "marker": "1b85c5b1-19d2-48a1-bb45-3ac75895aeed.4235.1",
    "index_type": "Normal",
    "owner": "joy",
    "ver": "0#3",
    "master_ver": "0#0",
    "mtime": "2019-03-19 22:02:50.726716",
    "max_marker": "0#",
    "usage": {
        "rgw.main": {
            "size": ,
            "size_actual": ,
            "size_utilized": ,
            "size_kb": ,
            "size_kb_actual": ,
            "size_kb_utilized": ,
            "num_objects":
        }
    },
    "bucket_quota": {
        "enabled": false,
        "check_on_raw": false,
        "max_size": -,
        "max_size_kb": ,
        "max_objects": -
    }
}

[root@ceph5 ~]# radosgw-admin bucket check --bucket=test

删除

[root@ceph5 ~]# radosgw-admin bucket rm --bucket=test

[root@ceph1 ceph]# s3cmd put --acl-public  /etc/ceph/ceph.conf  s3://test/ceph

upload: '/etc/ceph/ceph.conf' -> 's3://test/ceph'  [ of ]
  of    % in    0s    20.96 kB/s  done
Public URL of the object is: http://test.ceph5/ceph

[root@ceph1 ceph]# curl http://test.ceph5.lab.example.com/ceph

# Please do not change this file directly since it is managed by Ansible and will be overwritten

[global]
fsid = 35a91e48--4e96-a7ee-980ab989d20d

mon initial members = ceph2,ceph3,ceph4
mon host = 172.25.250.11,172.25.250.12,172.25.250.13

public network = 172.25.250.0/
cluster network = 172.25.250.0/

auth_cluster_required = cephx
auth_service_required = cephx
auth_client_required = cephx

[osd]
osd mkfs type = xfs
osd mkfs options xfs = -f -i size=
osd mount options xfs = noatime,largeio,inode64,swalloc
osd journal size = 

[mon]
mon_allow_pool_delete = true

[root@ceph1 ceph]# s3cmd get s3://test/demoobject ./demoobject

download: 's3://test/demoobject' -> './demoobject' [ of ]
 of  % in 0s 1346.20 B/s done

[root@ceph1 ceph]# cat ./demoobject

4.6 查看底层数据

[root@ceph5 ~]# ceph osd pool ls
rbd
rbdmirror
.rgw.root
default.rgw.control
default.rgw.meta
default.rgw.log
default.rgw.buckets.index
default.rgw.buckets.data
[root@ceph5 ~]#  rados -p  default.rgw.buckets.index ls --cluster backup
.dir.1b85c5b1-19d2-48a1-bb45-3ac75895aeed.4235.1
[root@ceph5 ~]#  rados -p  default.rgw.buckets.data ls
error opening pool default.rgw.buckets.data: () No such file or directory
[root@ceph5 ~]#  rados -p  default.rgw.buckets.data ls --cluster backup
1b85c5b1-19d2-48a1-bb45-3ac75895aeed..1_demoobject
1b85c5b1-19d2-48a1-bb45-3ac75895aeed..1_ceph

实验完成

---

博主声明：本文的内容来源主要来自誉天教育晏威老师，由本人实验完成操作验证，需要的博友请联系誉天教育（http://www.yutianedu.com/），获得官方同意或者晏老师（https://www.cnblogs.com/breezey/）本人同意即可转载，谢谢！