采用Operator-sdk轻松将helm chart转为Operator

去年就接触Operator,从Oracle发布的WebLogic Operator到mySQL Operator,构建的源码一大堆，但感觉一直缺少合适的开发框架能够避免复杂性快速生成，

随着技术的日益成熟，目前基于helm Operator轻松解决快速安装的问题，值得尝试一下。

下图是Operator框架的成熟度模型，基于不同的阶段，采用不同的技术满足全生命周期管理的需求

本文主要是以tomcat为例来进行快速的生成一个tomcat Operator.

1.环境准备

• 安装go

下载安装，参考 https://golang.org/doc/install?download=go1.11.5.linux-amd64.tar.gz，不再详诉。

• 安装编译环境

yum -y install gcc automake autoconf libtool make

• 安装Operator CLI

mkdir -p $GOPATH/src/github.com/operator-framework
cd $GOPATH/src/github.com/operator-framework
git clone https://github.com/operator-framework/operator-sdk
cd operator-sdk
git checkout master
make install

2.构建Tomcat Operator

• 新建项目

cd /usr/local/go/src/github.com/operator-framework/operator-sdk
operator-sdk new tomcat-operator --cluster-scoped --api-version=example.com/v1alpha1 --kind=Tomcat --type=helm

这里建立的是cluster-scope,意思是全集群都可以用，缺省是当前的命名空间，看一下有什么东西

[root@master operator-sdk]# tree tomcat-operator
tomcat-operator
├── build
│   └── Dockerfile
├── deploy
│   ├── crds
│   │   ├── example_v1alpha1_tomcat_crd.yaml
│   │   └── example_v1alpha1_tomcat_cr.yaml
│   ├── operator.yaml
│   ├── role_binding.yaml
│   ├── role.yaml
│   └── service_account.yaml
├── helm-charts
│   └── tomcat
│       ├── charts
│       ├── Chart.yaml
│       ├── templates
│       │   ├── deployment.yaml
│       │   ├── _helpers.tpl
│       │   ├── ingress.yaml
│       │   ├── NOTES.txt
│       │   ├── service.yaml
│       │   └── tests
│       │       └── test-connection.yaml
│       └── values.yaml
└── watches.yaml

• 客户化Operator的逻辑

主要是针对templates下面的deployment.yaml进行修改，因为缺省是以nginx为模板来做的，所以端口都是80,需要修改成8080

主要的修改就是deployment.yaml和values.yaml, 当然如果需要部署多个服务，同时多个服务由一定的依赖关系可以在helm中进行实现，镜像及版本的修改在values.yaml里面，我贴一段

[root@master tomcat]# cat values.yaml
# Default values for tomcat.
# This is a YAML-formatted file.
# Declare variables to be passed into your templates.

replicaCount: 

image:
  repository: registry.example.com/tomcat
  tag: -slim
  pullPolicy: IfNotPresent

nameOverride: ""
fullnameOverride: ""

service:
  type: ClusterIP
  port: 

ingress:
  enabled: false
  annotations: {}
    # kubernetes.io/ingress.class: nginx
    # kubernetes.io/tls-acme: "true"
  paths: []

我就修改了image和端口部分。

example_v1alpha1_tomcat_cr.yaml这个文件主要用于部署tomcat类型的实例的yaml文件，

可以修改deploy/crds/example_v1alpha1_tomcat_cr.yaml,部署多个实例

[root@master crds]# cat example_v1alpha1_tomcat_cr.yaml
apiVersion: example.com/v1alpha1
kind: Tomcat
metadata:
  name: example-tomcat
spec:
  # Default values copied from <project_dir>/helm-charts/tomcat/values.yaml

  # Default values for tomcat.
  # This is a YAML-formatted file.
  # Declare variables to be passed into your templates.

  replicaCount: 

  image:
    repository: registry.example.com/tomcat
    tag: -slim
    pullPolicy: IfNotPresent

  nameOverride: ""
  fullnameOverride: ""

打开可以看到，基本就是指定了Kind为Tomcat,下面具体的值都可以从values.yaml中拷贝，并且可以覆盖values.yaml的值。

3.部署Operator到集群

• 部署CRD

oc create -f deploy/crds/example_v1alpha1_nginx_crd.yaml

Openshift需要知道客户化的资源定义，这个定义就通过这个脚本，指定了watch.

• 生成Operator的镜像

首先部署的时候是基于build目录下的Dockerfile,因为访问不到，所以做了个跳转，把Dockerfile的quay.io/operator-framework/helm-operator:v0.5.0路径修改掉

[root@master build]# cat Dockerfile
FROM docker.io/ericnie2017/helm-operator:latest

COPY helm-charts/ ${HOME}/helm-charts/
COPY watches.yaml ${HOME}/watches.yaml

然后运行

operator-sdk build registry.example.com/example/tomcat-operator:v0.0.1

[root@master tomcat-operator]# operator-sdk build registry.example.com/example/tomcat-operator:v0.0.1
INFO[] Building Docker image registry.example.com/example/tomcat-operator:v0.0.1
Sending build context to Docker daemon 111.1 kB
Step / : FROM docker.io/ericnie2017/helm-operator:latest
 ---> f0d56774da3e
Step / : COPY helm-charts/ ${HOME}/helm-charts/
 ---> 9f77f7fba44d
Removing intermediate container efd44d601b0a
Step / : COPY watches.yaml ${HOME}/watches.yaml
 ---> 7469e31336af
Removing intermediate container 73189235ec15
Successfully built 7469e31336af
INFO[] Operator build complete.    

build语句会把我们定制的Operator生成一个镜像，运行完push到镜像仓库让全集群可以访问。

[root@master tomcat-operator]# docker push registry.example.com/example/tomcat-operator:v0.0.1
The push refers to a repository [registry.example.com/example/tomcat-operator]
ae10451a67a5: Pushed
bebcddc5922f: Pushed
e256e39f5897: Pushed
d724046711d4: Pushed
903dc29d7cf3: Pushed
e79522dce35e: Pushed
v0.0.1: digest: sha256:a439041a9de91f0fee04f4cd15c554d8a03ec37a286760415b015cbdce7f4315 size: 

• 建立相关的角色，权限和CRD的信息

执行几个sed操作用于对生成模板的替换操作。

[root@master tomcat-operator]# sed -i 's|REPLACE_IMAGE|registry.example.com/example/tomcat-operator:v0.0.1|g' deploy/operator.yaml
[root@master tomcat-operator]# oc config view --minify -o jsonpath='{.contexts[0].context.namespace}'
default

[root@master tomcat-operator]# sed -i "s|REPLACE_NAMESPACE|default|g" deploy/role_binding.yaml

一切就绪，开始创建

oc create -f deploy/service_account.yaml
oc create -f deploy/role.yaml
oc create -f deploy/role_binding.yaml
oc create -f deploy/operator.yaml

因为openshift自己的权限要求比较严格，干脆直接加成集群管理员了。

[root@master tomcat-operator]# oc adm policy add-cluster-role-to-user cluster-admin system:serviceaccount:default:tomcat-operator
cluster role "cluster-admin" added: "system:serviceaccount:default:tomcat-operator"

查看一下，已经创建起来了。这时这个Operator类型就作为一个Pod在容器内运行。

[root@master crds]# oc get deployment
NAME              DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
tomcat-operator                                            30m

4.创建和删除tomcat Operator实例

建立实例，先查看一下这个创建的yaml文件，Kind就是我们指定的类型Tomcat，而下面的值就是从values.yaml而来，可以覆盖也可以不用覆盖。

[root@master crds]# cat example_v1alpha1_tomcat_cr.yaml
apiVersion: example.com/v1alpha1
kind: Tomcat
metadata:
  name: example-tomcat
spec:
  # Default values copied from <project_dir>/helm-charts/tomcat/values.yaml

  # Default values for tomcat.
  # This is a YAML-formatted file.
  # Declare variables to be passed into your templates.

  replicaCount: 

  image:
    repository: registry.example.com/tomcat
    tag: -slim
    pullPolicy: IfNotPresent

  nameOverride: ""
  fullnameOverride: ""

  service:
    type: ClusterIP
    port: 

  ingress:
    enabled: false
    annotations: {}
      # kubernetes.io/ingress.class: nginx
      # kubernetes.io/tls-acme: "true"
    paths: []
    hosts:
      - chart-example.local
    tls: []
    #  - secretName: chart-example-tls
    #    hosts:
    #      - chart-example.local

  resources: {}
    # We usually recommend not to specify default resources and to leave this as a conscious
    # choice for the user. This also increases chances charts run on environments with little
    # resources, such as Minikube. If you do want to specify resources, uncomment the following
    # lines, adjust them as necessary, and remove the curly braces after 'resources:'.
    # limits:
    #  cpu: 100m
    #  memory: 128Mi
    # requests:
    #  cpu: 100m
    #  memory: 128Mi

  nodeSelector: {}
  tolerations: []

  affinity: {}
  

[root@master crds]# oc create  -f example_v1alpha1_tomcat_cr.yaml
tomcat.example.com/example-tomcat created
[root@master crds]# oc get pods
NAME                                                        READY     STATUS    RESTARTS   AGE
docker-registry--gl8jh                                     /       Running             18d
example-tomcat-1xvukmzvgn1tijep2w61xgm56-69457d7456-fm49d   /       Running             11s
example-tomcat-1xvukmzvgn1tijep2w61xgm56-69457d7456-twjhk   /       Running             11s
registry-console--6m4cq                                    /       Running             8d
router--7gx4b                                              /       Running             9d
tomcat-operator-75dc656956-hhnfd                            /       Running             27m

看到已经运行起来了，但是没有ready,没有ready的原因是readness和liveness的端口在deployment.yaml里面设置错了，没有修改成8080.

进去查看Pod日志，已经正常运行。

查看一下自定义对象，有一个example-tomcat，包含了2个pod

[root@master crds]# oc get Tomcat
NAME             AGE
example-tomcat   10s

删除实例

[root@master crds]# oc delete -f example_v1alpha1_tomcat_cr.yaml
tomcat.example.com "example-tomcat" deleted

好把，这是一个简单的Helm Operator上手的实验，随着Operator的成熟，在OpenShift 4.0版本中已经有很多组件化的部署都基于Operator来实现。