SUSE CaaS Platform 4 - 使用 Ceph RBD 作为持久存储（动态）

存储类

• 存储类（storage class）是kubernetes资源类型，它是由管理员为管理PV之便而按需创建的类别
• 存储类好处是支持 PV 的动态创建，系统按PVC的需求标准动态创建适配的PV会为存储管理带来极大的灵活性。
• PV的动态供给，其重点是在存储类的定义，其分类大概是对存储的性能进行分类的，如图1：金存储类、银存储类、铜存储类等。

图1 基于综合服务质量的存储系统分类

一、Provisioner（存储分配器）

Storage class 有一个分配器，用来决定使用哪个卷插件分配 PV。图2 中可以看到，目前 Ceph 只有 RBD 接口支持内部分配器。

  图2 各存储插件对动态供给方式的支持状况

二、实验环境搭建 - 动态供给

图3 实验环境架构图

环境设置

1、搭建环境

• 操作系统版本： SLES15 SP1，无需安装 swap
• 内核版本：4.12.14-197.18-default
• Kubernetes版本：v1.15.2
• VMware Workstation 14

2、虚拟化环境搭建和系统安装参考：

• SUSE Storage6 环境搭建详细步骤 - Win10 + VMware WorkStation
• SUSE Linux Enterprise 15 SP1 系统安装
• SUSE Ceph 快速部署 - Storage6
• SUSE CaaS Platform 4 - 安装部署

安装部署

1、所有 CaaS Platform 节点安装

# zypper install ceph-common

复制 ceph.conf 到 worker 节点上

# scp admin:/etc/ceph/ceph.conf /etc/ceph/ 

2、创建池,并将应用名称与存储池关联

# ceph osd pool create caasp4-dynamic
# ceph osd pool application enable caasp4-dynamic rbd 

3、创建 CaaSP4 client user

# cd /etc/ceph
# ceph auth get-or-create client.caasp4-dynamic mon 'allow r' \
    osd 'allow class-read object_prefix rbd_children, allow rwx pool=caasp4-dynamic' \
    -o ceph.client.dynamic.keyring

4、获取 client.admin 用户 key 信息，并生成基于 base64 编码 key

# ceph auth get client.admin
exported keyring for client.admin
[client.admin]
        key = AQA9w4VdAAAAABAAHZr5bVwkALYo6aLVryt7YA==
        caps mds = "allow *"
        caps mgr = "allow *"
        caps mon = "allow *"
        caps osd = "allow *

# echo AQA9w4VdAAAAABAAHZr5bVwkALYo6aLVryt7YA== | base64
QVFBOXc0VmRBQUFBQUJBQUhacjViVndrQUxZbzZhTFZyeXQ3WUE9PQo=

5、在Master节点上，为 client.admin，创建 secret 资源

# vim ceph-secret-admin.yaml
apiVersion: v1
kind: Secret
metadata:
  name: ceph-secret-admin
  namespace: kube-system
data:
  key: QVFBOXc0VmRBQUFBQUJBQUhacjViVndrQUxZbzZhTFZyeXQ3WUE9PQo=
type: kubernetes.io/rbd

# kubectl get secrets -n kube-system
NAME                                       TYPE                                 DATA   AGE
....
ceph-secret-admin                          kubernetes.io/rbd                           24s
....

6、获取 client.caasp4-dynamic 用户 key 信息，并生成基于 base64编码的key

# ceph auth get client.caasp4-dynamic
exported keyring for client.caasp4-dynamic
[client.caasp4-dynamic]
        key = AQA29ppdTDmzHhAAET2mSbvovrS67kspPlqmLA==
        caps mon = "allow r"
        caps osd = "allow class-read object_prefix rbd_children, allow rwx pool=caasp4-dynamic"

# echo AQA29ppdTDmzHhAAET2mSbvovrS67kspPlqmLA== | base64
QVFBMjlwcGRURG16SGhBQUVUMm1TYnZvdnJTNjdrc3BQbHFtTEE9PQo=

7、在Master节点上，为 client.caasp4-dynamic 创建 secret

# vim ceph-secret-user.yaml
apiVersion: v1
kind: Secret
metadata:
  name: ceph-user-secret
  namespace: default
data:
  key: QVFBMjlwcGRURG16SGhBQUVUMm1TYnZvdnJTNjdrc3BQbHFtTEE9PQo=
type: kubernetes.io/rbd

# kubectl create -f ceph-secret-user.yaml
secret "ceph-user-secret" created

# kubectl get secrets
NAME                  TYPE                                  DATA   AGE
ceph-secret-test      Opaque                                      20h
ceph-user-secret      kubernetes.io/rbd                           4s
default-token-4hslq   kubernetes.io/service-account-token         24h

8、创建 storage class 存储类

# vim ceph-storageclass.yaml
apiVersion: storage.k8s.io/v1beta1
kind: StorageClass
metadata:
  name: dynamic
  annotations:
     storageclass.beta.kubernetes.io/is-default-class: "true"
provisioner: kubernetes.io/rbd
parameters:
  monitors: 192.168.2.40:,192.168.2.41:,192.168.2.42:
  adminId: admin
  adminSecretName: ceph-secret-admin
  adminSecretNamespace: kube-system
  pool: caasp4-dynamic
  userId: caasp4-dynamic
  userSecretName: ceph-user-secret

# kubectl create -f ceph-storageclass.yaml
storageclass "dynamic" created

# kubectl get storageclasses
NAME                PROVISIONER         AGE
dynamic (default)   kubernetes.io/rbd   10m  

StorageClass SPEC

（1）provisioner(供给方)：即提供存储资源的存储系统，供给方名字都以“kubernetes.io”为前缀
（2）parameters（参数）：使用参数描述要关联到的存储卷，不同的provisioner有不同的参数

kubernetes 官方参数说明

9、创建 PVC

# vim ceph-pvc.yaml
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: ceph-claim-dynamic
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 3Gi

# kubectl create -f ceph-pvc.yaml
persistentvolumeclaim "ceph-claim-dynamic" created

# kubectl get pvc -o wide
NAME                 STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE   VOLUMEMODE
ceph-claim-dynamic   Bound    pvc-70b3b3ca---a898-c1f5524de008   3Gi        RWO            dynamic        10m   Filesystem

10、创建 Pod

vim ceph-pod-dynamic.yaml
apiVersion: v1
kind: Pod
metadata:
  name: ceph-pod1-dynamic
spec:
  containers:
  - name: ceph-busybox
    image: busybox
    command: ["sleep", ""]
    volumeMounts:
    - name: ceph-vol1-dynamic
      mountPath: /usr/share/busybox
      readOnly: false
  volumes:
  - name: ceph-vol1-dynamic
    persistentVolumeClaim:
      claimName: ceph-claim-dynamic

# kubectl create -f ceph-pod-dynamic.yaml
pod "ceph-pod1-dynamic" created

# kubectl get pods
NAME                READY   STATUS    RESTARTS   AGE
ceph-pod1-dynamic   /     Running             24m

# kubectl get pods -o wide
NAME                READY   STATUS    RESTARTS   AGE   IP             NODE       NOMINATED NODE   READINESS GATES
ceph-pod1-dynamic   /     Running             48m   10.244.2.194   worker02   <none>           <none>

# kubectl get pv
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                        STORAGECLASS   REASON   AGE
pvc-70b3b3ca-5267-4417-a898-c1f5524de008   3Gi        RWO            Delete           Bound    default/ceph-claim-dynamic   dynamic                 169m

11、storage6 分布式存储上，查看是否创建了镜像

admin:/etc/ceph # rbd ls -p caasp4-dynamic
kubernetes-dynamic-pvc-e6d98bbf-50e1--a9fc-867d1db810c8

admin:/etc/ceph # rbd info kubernetes-dynamic-pvc-e6d98bbf-50e1--a9fc-867d1db810c8 -p caasp4-dynamic
rbd image 'kubernetes-dynamic-pvc-e6d98bbf-50e1-4488-a9fc-867d1db810c8':
        size  GiB in  objects
        order  ( MiB objects)
        snapshot_count:
        id: cd53f75a03b15
        block_name_prefix: rbd_data.cd53f75a03b15
        format:
        features:
        op_features:
        flags:
        create_timestamp: Mon Oct   ::
        access_timestamp: Mon Oct   ::
        modify_timestamp: Mon Oct   :: 

12、worker02节点上，查看RBD映射

# rbd showmapped
id pool           namespace image                                                       snap device
0  caasp4-dynamic           kubernetes-dynamic-pvc-e6d98bbf-50e1-4488-a9fc-867d1db810c8 -    /dev/rbd0

# df -h | grep dev/rbd0
/dev/rbd0                2.9G  9.0M  2.9G   1% /var/lib/kubelet/pods/e7c75785-4533-4fac-b4ab-368c75e16421/volumes/kubernetes.io~rbd/pvc-70b3b3ca-5267-4417-a898-c1f5524de008

排错

1、搭建的时候，发现创建PVC的时候 pending 状态。

# kubectl get pvc
NAME                 STATUS    VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS   AGE
ceph-claim-dynamic   Pending                                      dynamic        44s

2、通过 event 事件查看，由于忘记创建 admin secret 导致，重新创建 admin secret 即可。

# kubectl get events
LAST SEEN   TYPE      REASON               OBJECT                                     MESSAGE
60s         Warning   ProvisioningFailed   persistentvolumeclaim/ceph-claim-dynamic   Failed to provision volume with StorageClass "dynamic": failed to get admin secret from ["kube-system"/"ceph-secret-admin"]: failed to get secret from ["kube-system"/"ceph-secret-admin"]: secrets "ceph-secret-admin" not found
35s         Warning   FailedScheduling     pod/ceph-pod1-dynamic                      pod has unbound immediate PersistentVolumeClaims (repeated 2 times)