kubernetes 1.17.2结合ceph13.2.8 实现jenkins部署并用traefik2.1代理

注：关于ceph、kubernetes集群的部署在此不声明，相信搜到本篇博文，你一定对ceph、kubernetes的部署环节手刃有余。

注：本篇博文牵扯到的技术点有：ceph、kubernetes、harbor、jenkins、traefik

ceph服务器操作

#ceph -s      //查看ceph集群状态
#ceph osd pool create jenkins 128      //创建pool  建议每个pool存放的是通类应用
#ceph auth get-or-create client.jenkins mon 'allow r' osd 'allow class-read, allow rwx pool=jenkins' -o ceph.client.jenkins.keyring    //创建普通用户管理对应pool

注意：ceph集群的状态要先调试成ok.

kubernetes拉取harbor镜像

# cat ~/.docker/config.json |base64 -w 0        //node节点访问私有仓库的认证
ewoJImF1dGhzIjogewoJCSJoYXJib3IubGludXguY29tIjogewoJCQkiYXV0aCI6ICJZV1J0YVc0NmVtbHpaV1psYVhwb2RRPT0iCgkJfQoJfSwKCSJIdHRwSGVhZGVycyI6IHsKCQkiVXNlci1BZ2VudCI6ICJEb2NrZXItQ2xpZW50LzE5LjAzLjUgKGxpbnV4KSIKCX0KfQ==
# cat secret_harbor.yaml
##########################################################################
#Author:                     zisefeizhu
#QQ:                         2********0
#Date:                       2020-02-19
#FileName:                   secret_harbor.yaml
#URL:                        https://www.cnblogs.com/zisefeizhu/
#Description:                The test script
#Copyright (C):              2020 All rights reserved
###########################################################################
apiVersion: v1
kind: Secret
metadata:
  name: k8s-harbor-login
type: kubernetes.io/dockerconfigjson
data:
  .dockerconfigjson: ewoJImF1dGhzIjogewoJCSJoYXJib3IubGludXguY29tIjogewoJCQkiYXV0aCI6ICJZV1J0YVc0NmVtbHpaV1psYVhwb2RRPT0iCgkJfQoJfSwKCSJIdHRwSGVhZGVycyI6IHsKCQkiVXNlci1BZ2VudCI6ICJEb2NrZXItQ2xpZW50LzE5LjAzLjUgKGxpbnV4KSIKCX0KfQ==
# kubectl create -f secret_harbor.yaml
secret/login created
# kubectl get secret
NAME                          TYPE                                  DATA   AGE
ceph-admin-secret             kubernetes.io/rbd                     1      3d16h
ceph-kube-secret              kubernetes.io/rbd                     1      3d16h

注意：前提是node节点可以访问到harbor，关于这部分可以参考我的这篇博文：https://www.cnblogs.com/zisefeizhu/p/12329864.html

部署动态存储

# pwd
/data/k8s/jenkins      //单个服务单个目录

# cat namespace.yaml    //单个服务单个名称空间，便于管理
##########################################################################
#Author:                     zisefeizhu
#QQ:                         2********0
#Date:                       2020-03-09
#FileName:                   namespace.yaml
#URL:                        https://www.cnblogs.com/zisefeizhu/
#Description:                The test script
#Copyright (C):              2020 All rights reserved
###########################################################################
apiVersion: v1
kind: Namespace
metadata:
  name: jenkins
  labels:
    name: jenkins

#kubernetes结合ceph需要使用第三方插件
# cat external-storage-rbd-provisioner.yaml
##########################################################################
#Author:                     zisefeizhu
#QQ:                         2********0
#Date:                       2020-03-09
#FileName:                   external-storage-rbd-provisioner.yaml
#URL:                        https://www.cnblogs.com/zisefeizhu/
#Description:                The test script
#Copyright (C):              2020 All rights reserved
###########################################################################
apiVersion: v1
kind: ServiceAccount
metadata:
  name: rbd-provisioner
  namespace: jenkins
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: rbd-provisioner
rules:
  - apiGroups: [""]
    resources: ["persistentvolumes"]
    verbs: ["get", "list", "watch", "create", "delete"]
  - apiGroups: [""]
    resources: ["persistentvolumeclaims"]
    verbs: ["get", "list", "watch", "update"]
  - apiGroups: ["storage.k8s.io"]
    resources: ["storageclasses"]
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources: ["events"]
    verbs: ["create", "update", "patch"]
  - apiGroups: [""]
    resources: ["endpoints"]
    verbs: ["get", "list", "watch", "create", "update", "patch"]
  - apiGroups: [""]
    resources: ["services"]
    resourceNames: ["kube-dns"]
    verbs: ["list", "get"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: rbd-provisioner
subjects:
  - kind: ServiceAccount
    name: rbd-provisioner
    namespace: jenkins
roleRef:
  kind: ClusterRole
  name: rbd-provisioner
  apiGroup: rbac.authorization.k8s.io

---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: rbd-provisioner
  namespace: jenkins
rules:
- apiGroups: [""]
  resources: ["secrets"]
  verbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: rbd-provisioner
  namespace: jenkins
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: rbd-provisioner
subjects:
- kind: ServiceAccount
  name: rbd-provisioner
  namespace: jenkins

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: rbd-provisioner
  namespace: jenkins
spec:
  replicas: 1
  selector:
    matchLabels:
      app: rbd-provisioner
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: rbd-provisioner
    spec:
      containers:
      - name: rbd-provisioner
        image: "harbor.linux.com/rbd/rbd-provisioner:latest"
        imagePullPolicy: IfNotPresent
        env:
        - name: PROVISIONER_NAME
          value: ceph.com/rbd
      imagePullSecrets:
        - name: k8s-harbor-login
      serviceAccount: rbd-provisioner

#敏感数据创建secret，这没什么可说的
# cat ceph-jenkins-secret.yaml
##########################################################################
#Author:                     zisefeizhu
#QQ:                         2********0
#Date:                       2020-03-09
#FileName:                   ceph-wordpress-secret.yaml
#URL:                        https://www.cnblogs.com/zisefeizhu/
#Description:                The test script
#Copyright (C):              2020 All rights reserved
###########################################################################
apiVersion: v1
kind: Secret
metadata:
  name: ceph-admin-secret
  namespace: jenkins
data:
  key: QVFBZ2pXVmVGOVJISkJBQTBTUDRoOTVZYVdHNEN6TzNaUWtIdVE9PQ==
type: kubernetes.io/rbd
---
apiVersion: v1
kind: Secret
metadata:
  name: ceph-jenkins-secret
  namespace: jenkins
data:
  key: QVFEUjRHWmVNUFJpRnhBQUQ1Zlg1UG9JRUNkMG85Qk5kVzN5SUE9PQ==
type: kubernetes.io/rbd

# cat ceph-jenkins-storageclass.yaml      //
##########################################################################
#Author:                     zisefeizhu
#QQ:                         2********0
#Date:                       2020-03-09
#FileName:                   ceph-wordpress-storageclass.yaml
#URL:                        https://www.cnblogs.com/zisefeizhu/
#Description:                The test script
#Copyright (C):              2020 All rights reserved
###########################################################################
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: ceph-jenkins
  namespace: jenkins
  annotations:
    storageclass.kubernetes.io/is-default-class: "false"
provisioner: ceph.com/rbd
reclaimPolicy: Retain
parameters:
  monitors: 20.0.0.207:6789,20.0.0.208:6789,20.0.0.210:6789
  adminId: admin
  adminSecretName: ceph-admin-secret
  adminSecretNamespace: jenkins
  pool: jenkins
  fsType: xfs
  userId: jenkins
  userSecretName: ceph-jenkins-secret
  imageFormat: "2"
  imageFeatures: "layering"

# cat jenkins-pvc.yaml
##########################################################################
#Author:                     zisefeizhu
#QQ:                         2********0
#Date:                       2020-03-10
#FileName:                   jenkins-pvc.yaml
#URL:                        https://www.cnblogs.com/zisefeizhu/
#Description:                The test script
#Copyright (C):              2020 All rights reserved
###########################################################################
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: jenkins-pvc
  namespace: jenkins
 # labels:
 #   app: gitlab
spec:
  storageClassName: ceph-jenkins
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 20Gi
# kubectl get pvc -n jenkins
NAME          STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
jenkins-pvc   Bound    pvc-d386c125-5302-468a-8a94-a2570f0a4ca0   20Gi       RWO            ceph-jenkins   2d8h

部署jenkins应用

# pwd
/data/k8s/jenkins

# cat jenkins.yaml     //核心资源清单
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: jenkins
  namespace: jenkins
  labels:
    name: jenkins
spec:
  selector:
    matchLabels:
      name: jenkins
  serviceName: jenkins
  replicas: 1
  updateStrategy:
    type: RollingUpdate
  template:
    metadata:
      name: jenkins
      labels:
        name: jenkins
    spec:
      terminationGracePeriodSeconds: 10
      serviceAccountName: jenkins
      #登陆私有仓库harbor认证
      imagePullSecrets:
        - name: k8s-harbor-login
      containers:
        - name: jenkins
          image: harbor.linux.com/dev/jenkins:lts
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 8080
            - containerPort: 50000
          resources:
            limits:
              cpu: 1
              memory: 1Gi
            requests:
              cpu: 0.5
              memory: 500Mi
          env:
            - name: LIMITS_MEMORY
              valueFrom:
                resourceFieldRef:
                  resource: limits.memory
                  divisor: 1Mi
            - name: JAVA_OPTS
              # value: -XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap -XX:MaxRAMFraction=1 -XshowSettings:vm -Dhudson.slaves.NodeProvisioner.initialDelay=0 -Dhudson.slaves.NodeProvisioner.MARGIN=50 -Dhudson.slaves.NodeProvisioner.MARGIN0=0.85
              value: -Xmx$(LIMITS_MEMORY)m -XshowSettings:vm -Dhudson.slaves.NodeProvisioner.initialDelay=0 -Dhudson.slaves.NodeProvisioner.MARGIN=50 -Dhudson.slaves.NodeProvisioner.MARGIN0=0.85
          volumeMounts:
            - name: jenkins-home
              mountPath: /var/jenkins_home
          livenessProbe:
            httpGet:
              path: /login
              port: 8080
            initialDelaySeconds: 60
            timeoutSeconds: 5
            failureThreshold: 12 # ~2 minutes
          readinessProbe:
            httpGet:
              path: /login
              port: 8080
            initialDelaySeconds: 60
            timeoutSeconds: 5
            failureThreshold: 12 # ~2 minutes
      securityContext:
        fsGroup: 1000
      volumes:
      - name: jenkins-home
        persistentVolumeClaim:
          claimName: jenkins-pvc
---
apiVersion: v1
kind: Service
metadata:
  name: jenkins
  namespace: jenkins
spec:
  # type: LoadBalancer
  selector:
    name: jenkins
  # ensure the client ip is propagated to avoid the invalid crumb issue when using LoadBalancer (k8s >=1.7)
  #externalTrafficPolicy: Local
  ports:
    -
      name: http
      port: 80
      targetPort: 8080
      protocol: TCP
    -
      name: agent
      port: 50000
      protocol: TCP

# cat jenkins-serviceaccount.yaml
##########################################################################
#Author:                     zisefeizhu
#QQ:                         2********0
#Date:                       2020-03-10
#FileName:                   jenkins-serviceaccount.yaml
#URL:                        https://www.cnblogs.com/zisefeizhu/
#Description:                The test script
#Copyright (C):              2020 All rights reserved
###########################################################################
# In GKE need to get RBAC permissions first with
# kubectl create clusterrolebinding cluster-admin-binding --clusterrole=cluster-admin [--user=<user-name>|--group=<group-name>]

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: jenkins
  namespace: jenkins
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: jenkins
  namespace: jenkins
rules:
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["create","delete","get","list","patch","update","watch"]
- apiGroups: [""]
  resources: ["pods/exec"]
  verbs: ["create","delete","get","list","patch","update","watch"]
- apiGroups: [""]
  resources: ["pods/log"]
  verbs: ["get","list","watch"]
- apiGroups: [""]
  resources: ["secrets"]
  verbs: ["get"]

---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: RoleBinding
metadata:
  name: jenkins
  namespace: jenkins
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: jenkins
subjects:
- kind: ServiceAccount
  name: jenkins
  namespace: jenkins
# kubectl get pods -n jenkins
NAME                               READY   STATUS        RESTARTS   AGE
jenkins-0                          1/1     Running       14         30h
rbd-provisioner-5c97b9d5ff-95qwj   1/1     Running       13         2d8h

部署代理

# cat jenkins-ingressroute.yaml
##########################################################################
#Author:                     zisefeizhu
#QQ:                         2********0
#Date:                       2020-03-10
#FileName:                   jenkins-ingressroute.yaml
#URL:                        https://www.cnblogs.com/zisefeizhu/
#Description:                The test script
#Copyright (C):              2020 All rights reserved
###########################################################################
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: jenkins
  namespace: jenkins
spec:
  entryPoints:
    - web
  routes:
  #登陆域名   需要自己在主机添加hosts解析  或者自建dns也行
  - match: Host(`jenkins.linux.com`)
    kind: Rule
    services:
    - name: jenkins    #和jenkins核心资源清单一致，  name是jenkins service的name
      port: 80         #jenkins pod 暴漏端口

登陆密码

# kubectl get pods -n jenkins
NAME                              READY   STATUS    RESTARTS   AGE
jenkins-0                         1/1     Running   1          18m
rbd-provisioner-dbc4c8b59-grfg2   1/1     Running   1          125m
# kubectl logs jenkins-0  -n jenkins
VM settings:
    Max. Heap Size: 1.00G
    Ergonomics Machine Class: server
    Using VM: OpenJDK 64-Bit Server VM

#注意 jenkins首次登陆密码
ac4fe3940ec145fe9104eda3ca390d0a    

This may also be found at: /var/jenkins_home/secrets/initialAdminPassword

*************************************************************
*************************************************************
*************************************************************

2020-03-10 02:44:33.446+0000 [id=39]	INFO	hudson.model.UpdateSite#updateData: Obtained the latest update center data file for UpdateSource default
2020-03-10 02:44:36.699+0000 [id=25]	INFO	hudson.model.UpdateSite#updateData: Obtained the latest update center data file for UpdateSource default
2020-03-10 02:44:42.206+0000 [id=25]	INFO	jenkins.InitReactorRunner$1#onAttained: Completed initialization
2020-03-10 02:44:43.044+0000 [id=39]	INFO	h.m.DownloadService$Downloadable#load: Obtained the updated data file for hudson.tasks.Maven.MavenInstaller
2020-03-10 02:44:43.044+0000 [id=39]	INFO	hudson.util.Retrier#start: Performed the action check updates server successfully at the attempt #1
2020-03-10 02:44:43.055+0000 [id=39]	INFO	hudson.model.AsyncPeriodicWork#lambda$doRun$0: Finished Download metadata. 81,778 ms
2020-03-10 02:44:43.414+0000 [id=19]	INFO	hudson.WebAppMain$3#run: Jenkins is fully up and running

登陆密码为：

ac4fe3940ec145fe9104eda3ca390d0a

注：本篇博文完全原创，后续将发布大量有技术的原创博文，请持续关注。

注：关于ceph集群的部署、kubernetes1.17.2高可用集群的部署可以看我的历史博文。

注：关于jenkins的使用不在本篇讲解范围内(主要是截图太多了，有空再发表吧)