构建gitlab+Jenkins+harbor+kubernetes的DevOps持续集成持续部署环境

整个环境的结构图。

一、准备工作

gitlab和harbor我是安装在kubernetes集群外的一台主机上的。

1.1、设置镜像源

docker-ce.repo

  1. [root@support harbor]# cat /etc/yum.repos.d/docker-ce.repo
  2. [docker-ce-stable]
  3. name=Docker CE Stable - $basearch
  4. baseurl=https://mirrors.aliyun.com/docker-ce/linux/centos/7/$basearch/stable
  5. enabled=1
  6. gpgcheck=1
  7. gpgkey=https://mirrors.aliyun.com/docker-ce/linux/centos/gpg
  9. [docker-ce-stable-debuginfo]
  10. name=Docker CE Stable - Debuginfo $basearch
  11. baseurl=https://mirrors.aliyun.com/docker-ce/linux/centos/7/debug-$basearch/stable
  12. enabled=0
  13. gpgcheck=1
  14. gpgkey=https://mirrors.aliyun.com/docker-ce/linux/centos/gpg
  16. [docker-ce-stable-source]
  17. name=Docker CE Stable - Sources
  18. baseurl=https://mirrors.aliyun.com/docker-ce/linux/centos/7/source/stable
  19. enabled=0
  20. gpgcheck=1
  21. gpgkey=https://mirrors.aliyun.com/docker-ce/linux/centos/gpg

1.2、安装依赖包

  1. [root@support yum.repos.d]# yum install -y docker-ce-18.09.7
  2. [root@support yum.repos.d]# yum install -y docker-compose
  3. [root@support yum.repos.d]# git
  4. [root@support yum.repos.d]# cat /etc/docker/daemon.json
  5. {"registry-mirrors": ["http://f1361db2.m.daocloud.io"]}
  6. [root@support yum.repos.d]# systemctl start docker

二、harbor部署

2.1、安装包

  1. [root@support yum.repos.d]# wget -b https://storage.googleapis.com/harbor-releases/release-1.9.0/harbor-offline-installer-v1.9.0.tgz
  2. Continuing in background, pid 9771.
  3. Output will be written to ‘wget-log’.
  4. [root@support ~]# tar zxf harbor-offline-installer-v1.9.0.tgz
  5. [root@support ~]# cd harbor
  6. [root@support harbor]# vi harbor.yml
  7. hostname: 139.9.134.177
  8. http:
  9.   port: 8080

2.2、部署

  1. [root@support harbor]# ./prepare 
  3. [root@support harbor]# ./install.sh 
  5. [root@support harbor]# docker-compose ps
  6.       Name                     Command              State             Ports
  7. -------------------------------------------------------------------------------------
  8. harbor-core         /harbor/harbor_core             Up
  9. harbor-db           /docker-entrypoint.sh           Up      5432/tcp
  10. harbor-jobservice   /harbor/harbor_jobservice       Up
  11.                     ...
  12. harbor-log          /bin/sh -c /usr/local/bin/      Up      127.0.0.1:1514->10514/tcp
  13.                     ...
  14. harbor-portal       nginx -g daemon off;            Up      8080/tcp
  15. nginx               nginx -g daemon off;            Up      0.0.0.0:8080->8080/tcp
  16. redis               redis-server /etc/redis.conf    Up      6379/tcp
  17. registry            /entrypoint.sh /etc/regist      Up      5000/tcp
  18.                     ...
  19. registryctl         /harbor/start.sh                Up

三、gitlab部署

3.1、拉取镜像

  1. [root@support yum.repos.d]# docker pull gitlab/gitlab-ce
  2. Using default tag: latest
  3. latest: Pulling from gitlab/gitlab-ce
  4. 16c48d79e9cc: Pull complete
  5. 3c654ad3ed7d: Pull complete
  6. 6276f4f9c29d: Pull complete
  7. a4bd43ad48ce: Pull complete
  8. 075ff90164f7: Pull complete
  9. 8ed147de678c: Pull complete
  10. c6b08aab9197: Pull complete
  11. 6c15d9b5013c: Pull complete
  12. de3573fbdb09: Pull complete
  13. 4b6e8211dc80: Verifying Checksum
  14. latest: Pulling from gitlab/gitlab-ce
  15. 16c48d79e9cc: Pull complete
  16. 3c654ad3ed7d: Pull complete
  17. 6276f4f9c29d: Pull complete
  18. a4bd43ad48ce: Pull complete
  19. 075ff90164f7: Pull complete
  20. 8ed147de678c: Pull complete
  21. c6b08aab9197: Pull complete
  22. 6c15d9b5013c: Pull complete
  23. de3573fbdb09: Pull complete
  24. 4b6e8211dc80: Pull complete
  25. Digest: sha256:eee5fc2589f9aa3cd4c1c1783d5b89667f74c4fc71c52df54660c12cc493011b
  26. Status: Downloaded newer image for gitlab/gitlab-ce:latest
  27. docker.io/gitlab/gitlab-ce:latest
  28. [root@support yum.repos.d]#

3.2、启动容器

  1. [root@bogon /]# docker run --detach \
  2. --hostname 139.9.134.177 \
  3. --publish 10443:443 --publish 10080:80 --publish 10022:22 \
  4. --name gitlab \
  5. --restart always \
  6. --volume /opt/gitlab/config:/etc/gitlab \
  7. --volume /opt/gitlab/logs:/var/log/gitlab \
  8. --volume /opt/gitlab/data:/var/opt/gitlab \
  9. gitlab/gitlab-ce:latest
  1. git仓库初始化
  2. git init --bare
  3. git clone
  1. yum install jenkins -y
  2. java -version
  4. tail -f /var/log/jenkins/jenkins.log
  5. log中输出jenkins网页端初始化密码。

四、jenkins部署

github上的kubernetes集群部署 jenkins

https://github.com/jenkinsci/kubernetes-plugin/blob/master/src/main/kubernetes/jenkins.yml

4.1、NFS-PV动态供给

NFS服务准备

  1. # yum安装nfs-utils
  2. [root@support ~]# yum install -y nfs-utils
  3. [root@support ~]# mkdir /ifs/kubernetes
  4. [root@support ~]# cat /etc/exports
  5. # 提供共享目录给10.0.0.0网段主机
  6. /ifs/kubernetes 10.0.0.0/24(rw,no_root_squash)
  7. [root@support ~]# systemctl start nfs
  8. [root@support ~]# exportfs -arv
  9. exporting 10.0.0.0/24:/ifs/kubernetes

nfs.yaml

  1. [root@master jenkins]# cat nfs.yaml
  2. kind: ClusterRole
  3. apiVersion: rbac.authorization.k8s.io/v1
  4. metadata:
  5.   name: nfs-client-provisioner-runner
  6. rules:
  7.   - apiGroups: [""]
  8.     resources: ["persistentvolumes"]
  9.     verbs: ["get", "list", "watch", "create", "delete"]
  10.   - apiGroups: [""]
  11.     resources: ["persistentvolumeclaims"]
  12.     verbs: ["get", "list", "watch", "update"]
  13.   - apiGroups: ["storage.k8s.io"]
  14.     resources: ["storageclasses"]
  15.     verbs: ["get", "list", "watch"]
  16.   - apiGroups: [""]
  17.     resources: ["events"]
  18.     verbs: ["create", "update", "patch"]
  20. ---
  22. kind: ClusterRoleBinding
  23. apiVersion: rbac.authorization.k8s.io/v1
  24. metadata:
  25.   name: run-nfs-client-provisioner
  26. subjects:
  27.   - kind: ServiceAccount
  28.     name: nfs-client-provisioner
  29.     namespace: default
  30. roleRef:
  31.   kind: ClusterRole
  32.   name: nfs-client-provisioner-runner
  33.   apiGroup: rbac.authorization.k8s.io
  35. ---
  37. kind: Role
  38. apiVersion: rbac.authorization.k8s.io/v1
  39. metadata:
  40.   name: leader-locking-nfs-client-provisioner
  41. rules:
  42.   - apiGroups: [""]
  43.     resources: ["endpoints"]
  44.     verbs: ["get", "list", "watch", "create", "update", "patch"]
  46. ---
  48. kind: RoleBinding
  49. apiVersion: rbac.authorization.k8s.io/v1
  50. metadata:
  51.   name: leader-locking-nfs-client-provisioner
  52. subjects:
  53.   - kind: ServiceAccount
  54.     name: nfs-client-provisioner
  55.     # replace with namespace where provisioner is deployed
  56.     namespace: default
  57. roleRef:
  58.   kind: Role
  59.   name: leader-locking-nfs-client-provisioner
  60.   apiGroup: rbac.authorization.k8s.io
  62. ---
  64. kind: StorageClass
  65. apiVersion: storage.k8s.io/v1
  66. metadata:
  67.   name: managed-nfs-storage
  68. provisioner: fuseim.pri/ifs # or choose another name, must match deployment's env PROVISIONER_NAME'
  69. parameters:
  70.   archiveOnDelete: "true"
  72. ---
  74. kind: ServiceAccount
  75. apiVersion: v1
  76. metadata:
  77.   name: nfs-client-provisioner
  79. ---
  81. kind: Deployment
  82. apiVersion: apps/v1
  83. metadata:
  84.   name: nfs-client-provisioner
  85. spec:
  86.   replicas: 1
  87.   strategy:
  88.     type: Recreate
  89.   selector:
  90.     matchLabels:
  91.       app: nfs-client-provisioner
  92.   template:
  93.     metadata:
  94.       labels:
  95.         app: nfs-client-provisioner
  96.     spec:
  97.       serviceAccountName: nfs-client-provisioner
  98.       containers:
  99.         - name: nfs-client-provisioner
  100.           image: lizhenliang/nfs-client-provisioner:latest
  101.           volumeMounts:
  102.             - name: nfs-client-root
  103.               mountPath: /persistentvolumes
  104.           env:
  105.             - name: PROVISIONER_NAME
  106.               value: fuseim.pri/ifs
  107.             - name: NFS_SERVER
  108.               value: 10.0.0.123
  109.             - name: NFS_PATH
  110.               value: /ifs/kubernetes
  111.       volumes:
  112.         - name: nfs-client-root
  113.           nfs:
  114.             server: 10.0.0.123
  115.             path: /ifs/kubernetes
  116. [root@master jenkins]#
  1. # 创建PV动态供给
  2. root@master jenkins]# kubectl apply -f nfs.yaml

4.2、Jenkins在kubernetes上部署

jenkins-master调度到K8S的master节点。

jenkins.yaml

  1. [root@master jenkins]# cat jenkins.yaml
  2. apiVersion: v1
  3. kind: Service
  4. metadata:
  5.   name: jenkins
  6. spec:
  7.   selector:
  8.     name: jenkins
  9.   type: NodePort
  10.   ports:
  11.     -
  12.       name: http
  13.       port: 80
  14.       targetPort: 8080
  15.       protocol: TCP
  16.       nodePort: 30006
  17.     -
  18.       name: agent
  19.       port: 50000
  20.       protocol: TCP
  22. ---
  24. apiVersion: v1
  25. kind: ServiceAccount
  26. metadata:
  27.   name: jenkins
  29. ---
  31. kind: Role
  32. apiVersion: rbac.authorization.k8s.io/v1
  33. metadata:
  34.   name: jenkins
  35. rules:
  36. - apiGroups: [""]
  37.   resources: ["pods"]
  38.   verbs: ["create","delete","get","list","patch","update","watch"]
  39. - apiGroups: [""]
  40.   resources: ["pods/exec"]
  41.   verbs: ["create","delete","get","list","patch","update","watch"]
  42. - apiGroups: [""]
  43.   resources: ["pods/log"]
  44.   verbs: ["get","list","watch"]
  45. - apiGroups: [""]
  46.   resources: ["secrets"]
  47.   verbs: ["get"]
  49. ---
  51. apiVersion: rbac.authorization.k8s.io/v1
  52. kind: RoleBinding
  53. metadata:
  54.   name: jenkins
  55. roleRef:
  56.   apiGroup: rbac.authorization.k8s.io
  57.   kind: Role
  58.   name: jenkins
  59. subjects:
  60. - kind: ServiceAccount
  61.   name: jenkins
  63. ---
  65. apiVersion: apps/v1
  66. kind: StatefulSet
  67. metadata:
  68.   name: jenkins
  69.   labels:
  70.     name: jenkins
  71. spec:
  72.   serviceName: jenkins
  73.   replicas: 1
  74.   updateStrategy:
  75.     type: RollingUpdate
  76.   selector:
  77.     matchLabels:
  78.       name: jenkins
  79.   template:
  80.     metadata:
  81.       name: jenkins
  82.       labels:
  83.         name: jenkins
  84.     spec:
  85.       terminationGracePeriodSeconds: 10
  86.       serviceAccountName: jenkins
  87.       # 调度到主节点上
  88.       nodeSelector:
  89.         labelName: master
  90.       # 容忍主节点污点
  91.       tolerations:
  92.       - key: node-role.kubernetes.io/master
  93.         effect: NoSchedule
  94.       containers:
  95.         - name: jenkins
  96.           image: jenkins/jenkins:lts-alpine
  97.           imagePullPolicy: Always
  98.           ports:
  99.             - containerPort: 8080
  100.             - containerPort: 50000
  101.           env:
  102.             - name: LIMITS_MEMORY
  103.               valueFrom:
  104.                 resourceFieldRef:
  105.                   resource: limits.memory
  106.                   divisor: 1Mi
  107.             - name: JAVA_OPTS
  108.               value: -Xmx$(LIMITS_MEMORY)m -XshowSettings:vm -Dhudson.slaves.NodeProvisioner.initialDelay=0 -Dhudson.slaves.NodeProvisioner.MARGIN=50 -Dhudson.slaves.NodeProvisioner.MARGIN0=0.85
  109.           volumeMounts:
  110.             - name: jenkins-home
  111.               mountPath: /var/jenkins_home
  112.           livenessProbe:
  113.             httpGet:
  114.               path: /login
  115.               port: 8080
  116.             initialDelaySeconds: 60
  117.             timeoutSeconds: 5
  118.             failureThreshold: 12
  119.           readinessProbe:
  120.             httpGet:
  121.               path: /login
  122.               port: 8080
  123.             initialDelaySeconds: 60
  124.             timeoutSeconds: 5
  125.             failureThreshold: 12
  126.       securityContext:
  127.         fsGroup: 1000
  128.   volumeClaimTemplates:
  129.   - metadata:
  130.       name: jenkins-home
  131.     spec:
  132.       storageClassName: "managed-nfs-storage"
  133.       accessModes: [ "ReadWriteOnce" ]
  134.       resources:
  135.         requests:
  136.           storage: 1Gi
  1. # 创建jenkins Pod
  2. root@master jenkins]# kubectl apply -f jenkins.yaml
  4. # 打开浏览器访问jenkins地址
  5. http://139.9.139.49:30006/
  7. # 卡在启动界面好久
  8. [root@support default-jenkins-home-jenkins-0-pvc-ea84462f-241e-4d38-a408-e07a59d4bf0e]# cat hudson.model.UpdateCenter.xml
  9. <?xml version='1.1' encoding='UTF-8'?>
  10. <sites>
  11.   <site>
  12.     <id>default</id>
  13.     <url>http://mirror.xmission.com/jenkins/updates/update-center.json</url>
  14.   </site>
  15. </sites>

4.3、插件安装

在jenkins中安装插件 系统管理 --> 插件管理

4.3.1、需要下载的插件列表

  1. Git plugin        git
  2. GitLab Plugin     gitlab
  3. Kubernetes plugin 动态创建代理
  4. Pipeline          流水线
  5. Email Extension   邮件扩展

安装插件实在太慢。几kb每秒 ╮(￣▽￣)╭

我们有一个思路解决这个问题 [](￣▽￣)*

4.3.2、告诉jenkins 我哪些插件需要更新

使用清华大学镜像地址https://mirrors.tuna.tsinghua.edu.cn/jenkins/updates/update-center.json

1.进入jenkins系统管理

2.进入插件管理（Manage Plugins）

-- > 高级 -- > 升级站点

4.3.3、原理

https://mirrors.tuna.tsinghua.edu.cn/jenkins/updates/update-center.json 这个文件里面 包含了所有插件的更新地址，清华把这个文件拿过来了，但是没有把里面的插件升级地址改成清华。下载插件还是要到国外主机去下载，这样只会获取更新信息快，实际下载插件慢的一批。

  1. curl -vvvv  http://updates.jenkins-ci.org/download/plugins/ApicaLoadtest/1.10/ApicaLoadtest.hpi
  2. 302到
  3. http://mirrors.jenkins-ci.org/plugins/ApicaLoadtest/1.10/ApicaLoadtest.hpi
  4. 又重定向到一个ftp地址分流。
  6. 清华的地址是：
  7. https://mirrors.tuna.tsinghua.edu.cn/jenkins/plugins/ApicaLoadtest/1.10/ApicaLoadtest.hpi
  8. 只要把mirrors.jenkins-ci.org 代理到 mirrors.tuna.tsinghua.edu.cn/jenkins 即可。

4.3.4、欺骗jenkins去清华下载插件

绑定 mirrors.jenkins-ci.org 域名到本机 /etc/hosts 中

  1. [root@support nginx]# cat /etc/hosts
  2. 127.0.0.1 mirrors.jenkins-ci.org

nginx反向代理至清华的jenkins插件下载地址

  1. [root@support ~]# cat /etc/nginx/nginx.conf
  2. user nginx;
  3. worker_processes auto;
  4. error_log /var/log/nginx/error.log;
  5. pid /run/nginx.pid;
  7. include /usr/share/nginx/modules/*.conf;
  9. events {
  10.     worker_connections 1024;
  11. }
  13. http {
  15.     access_log  /var/log/nginx/access.log;
  17.     sendfile            on;
  18.     tcp_nopush          on;
  19.     tcp_nodelay         on;
  20.     keepalive_timeout   65;
  21.     types_hash_max_size 2048;
  23.     include             /etc/nginx/mime.types;
  24.     default_type        application/octet-stream;
  26.     server
  27.     {
  28.         listen 80;
  29.         server_name mirrors.jenkins-ci.org;
  30.         root    /usr/share/nginx/html;
  32.         location / {
  33.             proxy_redirect off;
  34.             proxy_pass https://mirrors.tuna.tsinghua.edu.cn/jenkins/;
  35.             proxy_set_header X-Real-IP $remote_addr;
  36.             proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
  37.             proxy_set_header Accept-Encoding "";
  38.             proxy_set_header Accept-Language "zh-CN";
  39.         }
  40.         index index.html index.htm index.php;
  42.         location ~ /\.
  43.         {
  44.             deny all;
  45.         }
  47.     }
  49. }

最后我们来看一下nginx访问日志。从本机发送的jenkins下载插件的请求全部转发到清华镜像源了。

  1. 127.0.0.1 - - [14/Oct/2019:23:40:32 +0800] "GET /plugins/kubernetes-credentials/0.4.1/kubernetes-credentials.hpi HTTP/1.1" 200 17893 "-" "Java/1.8.0_222"
  2. 127.0.0.1 - - [14/Oct/2019:23:40:37 +0800] "GET /plugins/variant/1.3/variant.hpi HTTP/1.1" 200 10252 "-" "Java/1.8.0_222"
  3. 127.0.0.1 - - [14/Oct/2019:23:40:40 +0800] "GET /plugins/kubernetes-client-api/4.6.0-2/kubernetes-client-api.hpi HTTP/1.1" 200 11281634 "-" "Java/1.8.0_222"
  4. 127.0.0.1 - - [14/Oct/2019:23:40:42 +0800] "GET /plugins/kubernetes/1.20.0/kubernetes.hpi HTTP/1.1" 200 320645 "-" "Java/1.8.0_222"
  5. 127.0.0.1 - - [14/Oct/2019:23:40:45 +0800] "GET /plugins/git/3.12.1/git.hpi HTTP/1.1" 200 2320552 "-" "Java/1.8.0_222"
  6. 127.0.0.1 - - [14/Oct/2019:23:40:47 +0800] "GET /plugins/gitlab-plugin/1.5.13/gitlab-plugin.hpi HTTP/1.1" 200 8456411 "-" "Java/1.8.0_222"

按照推荐做法，发现速度太快了，基本上秒下 (￣ˇ￣) 网上的大部分教程只做到第一步，设置完了，有时候能加速，有时候不能，这才是真正的最终解决方案。

当然为了做到这一步踩了一晚上的坑，首先在K8S中以pod部署的jenkins不能用这种代理方式。在苦试无果后，我只能非常粗暴的在NFS服务器上安装了一个同版本的jenkins，实测发现pod中的本地持久目录/var/jenkins_home所对应的路径中的文件直接拷贝至/var/lib/jenkins中，这个新jenkins的运行状态与pod中的jenkins一致。所以在新jenkins下载插件后，将插件目录/var/lib/jenkins/plugins直接拷贝进pod持久卷即可。

4.4、gitlab触发jenkins

4.4.1、gitlab生成token

复制此token，此token只显示一次：vze6nS8tLAQ1dVpdaHYU

4.4.2、jenkins配置连接gitlab

点击 系统管理 --> 系统设置，找到gitlab

类型选择gitlab api token，将gitab生成的token填入

4.4.3、创建jenkins任务

这个地址用来设置gitlab的webhook：http://139.9.139.49:30006/project/gitlab-citest-pipeline

点击生成token：2daf58bf638f04ce9e201ef0df9bec0f

此token也是用来设置gitlab的webhook

4.4.4、gitlab设置webhooks

4.4.5、提交代码至gitlab触发jenkins任务

先将gitlab上面的仓库克隆至本地

  1. [root@support ~]# git clone http://139.9.134.177:10080/miao/citest.git
  2. Cloning into 'citest'...
  3. remote: Enumerating objects: 3, done.
  4. remote: Counting objects: 100% (3/3), done.
  5. remote: Total 3 (delta 0), reused 0 (delta 0)
  6. Unpacking objects: 100% (3/3), done.

修改后提交代码至gitlab

  1. [root@support citest]# git commit -m "Testing gitlab and jenkins Connection #1"
  2. [master 03264a7] Testing gitlab and jenkins Connection 1
  3.  1 file changed, 3 insertions(+), 1 deletion(-)
  4. [root@support citest]# git push origin master
  5. Username for 'http://139.9.134.177:10080': miao
  6. Password for 'http://miao@139.9.134.177:10080':
  7. Counting objects: 5, done.
  8. Writing objects: 100% (3/3), 294 bytes | 0 bytes/s, done.
  9. Total 3 (delta 0), reused 0 (delta 0)
  10. To http://139.9.134.177:10080/miao/citest.git
  11.    25f05bb..03264a7  master -> master

jenkins任务已经开始执行

显示任务由gitlab触发，第一阶段成功。

4.5、jenkins在kubernetes中创建动态代理

我们这里使用了Docker in Docker技术，就是把jenkins部署在k8s里。jenkins master会动态创建slave pod，使用slave pod运行代码克隆，项目构建，镜像构建等指令操作。构成完成以后删除这个slave pod。减轻jenkins-master的负载，可以极大地提高资源利用率。

4.5.1、配置连接kubernetes

我们已经安装了Kubernetes插件，我们直接在jenkins中点击

系统管理 -- > 系统设置 -- > 拉到最底下有一个云。

新增一个云 --> kubernetes

因为jenkins是直接运行在k8s上的，所以可以直接通过k8s的dns访问kubernetes的service名称的。点击 --> 测试连接，成功连接k8s。

然后点击-->保存

4.5.2、构建Jenkins-Slave镜像

github官方构建slave文档

https://github.com/jenkinsci/docker-jnlp-slave

构建jenkins-slave镜像我们需要准备四个文件

1、在jenkins地址栏输入下列地址获取slave.jar

http://119.3.226.210:30006/jnlpJars/slave.jar

2、slave.jar的启动脚本jenkins-slave

  1. [root@support jenkins-slave]# cat jenkins-slave
  2. #!/usr/bin/env sh
  4. if [ $# -eq 1 ]; then
  6. 	# if `docker run` only has one arguments, we assume user is running alternate command like `bash` to inspect the image
  7. 	exec "$@"
  9. else
  11. 	# if -tunnel is not provided try env vars
  12. 	case "$@" in
  13. 		*"-tunnel "*) ;;
  14. 		*)
  15. 		if [ ! -z "$JENKINS_TUNNEL" ]; then
  16. 			TUNNEL="-tunnel $JENKINS_TUNNEL"
  17. 		fi ;;
  18. 	esac
  20. 	# if -workDir is not provided try env vars
  21. 	if [ ! -z "$JENKINS_AGENT_WORKDIR" ]; then
  22. 		case "$@" in
  23. 			*"-workDir"*) echo "Warning: Work directory is defined twice in command-line arguments and the environment variable" ;;
  24. 			*)
  25. 			WORKDIR="-workDir $JENKINS_AGENT_WORKDIR" ;;
  26. 		esac
  27. 	fi
  29. 	if [ -n "$JENKINS_URL" ]; then
  30. 		URL="-url $JENKINS_URL"
  31. 	fi
  33. 	if [ -n "$JENKINS_NAME" ]; then
  34. 		JENKINS_AGENT_NAME="$JENKINS_NAME"
  35. 	fi  
  37. 	if [ -z "$JNLP_PROTOCOL_OPTS" ]; then
  38. 		echo "Warning: JnlpProtocol3 is disabled by default, use JNLP_PROTOCOL_OPTS to alter the behavior"
  39. 		JNLP_PROTOCOL_OPTS="-Dorg.jenkinsci.remoting.engine.JnlpProtocol3.disabled=true"
  40. 	fi
  42. 	# If both required options are defined, do not pass the parameters
  43. 	OPT_JENKINS_SECRET=""
  44. 	if [ -n "$JENKINS_SECRET" ]; then
  45. 		case "$@" in
  46. 			*"${JENKINS_SECRET}"*) echo "Warning: SECRET is defined twice in command-line arguments and the environment variable" ;;
  47. 			*)
  48. 			OPT_JENKINS_SECRET="${JENKINS_SECRET}" ;;
  49. 		esac
  50. 	fi
  52. 	OPT_JENKINS_AGENT_NAME=""
  53. 	if [ -n "$JENKINS_AGENT_NAME" ]; then
  54. 		case "$@" in
  55. 			*"${JENKINS_AGENT_NAME}"*) echo "Warning: AGENT_NAME is defined twice in command-line arguments and the environment variable" ;;
  56. 			*)
  57. 			OPT_JENKINS_AGENT_NAME="${JENKINS_AGENT_NAME}" ;;
  58. 		esac
  59. 	fi
  61. 	#TODO: Handle the case when the command-line and Environment variable contain different values.
  62. 	#It is fine it blows up for now since it should lead to an error anyway.
  64. 	exec java $JAVA_OPTS $JNLP_PROTOCOL_OPTS -cp /usr/share/jenkins/slave.jar hudson.remoting.jnlp.Main -headless $TUNNEL $URL $WORKDIR $OPT_JENKINS_SECRET $OPT_JENKINS_AGENT_NAME "$@"
  65. fi

3、maven的配置文件

  1. [root@support jenkins-slave]# cat settings.xml
  2. <?xml version="1.0" encoding="UTF-8"?>
  3. <settings xmlns="http://maven.apache.org/SETTINGS/1.0.0"
  4.           xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  5.           xsi:schemaLocation="http://maven.apache.org/SETTINGS/1.0.0 http://maven.apache.org/xsd/settings-1.0.0.xsd">
  6.   <pluginGroups>
  7.   </pluginGroups>
  8.   <proxies>
  9.   </proxies>
  10.   <servers>
  11.   </servers>
  12.   <mirrors>
  13.     <mirror>
  14.       <id>central</id>
  15.       <mirrorOf>central</mirrorOf>
  16.       <name>aliyun maven</name>
  17.       <url>https://maven.aliyun.com/repository/public</url>
  18.     </mirror>
  19.   </mirrors>
  20.   <profiles>
  21.   </profiles>
  22. </settings>

4、Dockerfile

  1. FROM centos:7
  2. LABEL maintainer lizhenliang
  4. # 使镜像具有拖git仓库，编译java代码的能力
  5. RUN yum install -y java-1.8.0-openjdk maven curl git libtool-ltdl-devel && \
  6.     yum clean all && \
  7.     rm -rf /var/cache/yum/* && \
  8.     mkdir -p /usr/share/jenkins
  10. # 将获取到slave.jar放入镜像
  11. COPY slave.jar /usr/share/jenkins/slave.jar
  12. # jenkins-slave执行脚本
  13. COPY jenkins-slave /usr/bin/jenkins-slave
  14. # settings.xml中设置了aliyun的镜像
  15. COPY settings.xml /etc/maven/settings.xml
  16. RUN chmod +x /usr/bin/jenkins-slave
  18. ENTRYPOINT ["jenkins-slave"]

把这4个文件放在同级目录下，接下来我们开始构建slave镜像

构建镜像并打上标签

  1. [root@support jenkins-slave]# docker build . -t 139.9.134.177:8080/jenkinsci/jenkins-slave-jdk:1.8
  2. [root@support jenkins-slave]# docker image ls
  3. REPOSITORY                                       TAG                        IMAGE ID            CREATED             SIZE
  4. 139.9.134.177:8080/jenkinsci/jenkins-slave-jdk   1.8                        940e56848837        3 minutes ago       535MB

开始推送镜像

http登录拒绝，docker默认是https的，需要修改daemon.json

  1. [root@support jenkins-slave]# docker login 139.9.134.177:8080
  2. Username: admin
  3. Password:
  4. Error response from daemon: Get https://139.9.134.177:8080/v2/: http: server gave HTTP response to HTTPS client
  5. # 增加http的信任
  6. [root@support ~]# cat /etc/docker/daemon.json
  7. {
  8.     "registry-mirrors": ["http://f1361db2.m.daocloud.io"],
  9.     "insecure-registries": ["http://139.9.134.177:8080"]
  10. }
  11. # 成功登录
  12. [root@support ~]# docker login 139.9.134.177:8080
  13. Username: admin
  14. Password:
  15. WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
  16. Configure a credential helper to remove this warning. See
  17. https://docs.docker.com/engine/reference/commandline/login/#credentials-store
  19. Login Succeeded

所有的k8s主机也需要配置访问harbor的地址。重启docker服务。

我们设置信任的地址为内网地址，以保证足够的速度。

4.5.3、Jenkins任务由k8s的pod执行

使用以下pipeline脚本动态创建pod

  1. // 镜像仓库地址
  2. def registry = "10.0.0.123:8080"
  4. podTemplate(label: 'jenkins-agent', cloud: 'kubernetes',
  5.     containers: [
  6.     containerTemplate(
  7.         name: 'jnlp',
  8.         image: "${registry}/jenkinsci/jenkins-slave-jdk:1.8"
  9.     )],
  10.     volumes: [
  11.         hostPathVolume(mountPath: '/var/run/docker.sock', hostPath: '/var/run/docker.sock'),
  12.         hostPathVolume(mountPath: '/usr/bin/docker', hostPath: '/usr/bin/docker')
  13.     ])
  14. {
  15.   node("jenkins-agent"){
  16.         stage('拉取代码') { // for display purposes
  17.             git 'http://139.9.134.177:10080/miao/citest.git'
  18.             sh 'ls'
  19.         }
  20.         stage('代码编译') {
  21.             echo 'ok'
  22.         }
  23.         stage('部署') {
  24.             echo 'ok'
  25.         }
  26.     }
  27. }

4.6、使用pipeline脚本持续集成

使用pipeline脚本将每次提交gitlab的代码拉取下来，编译为docker镜像推送至harbor中。

在这里我们需要先配置两个凭据，因为我们gitlab代码仓库是私有的，harbor仓库也是私有的，只有配置凭据jenkins才能访问。

输入gitlab的账号和密码，生成一个凭据后，复制凭据的id，在pipeline中引用

输入harbor的账号和密码，生成一个凭据后，复制凭据的id，在pipeline中引用

  1. // 镜像仓库地址
  2. def registry = "10.0.0.123:8080"
  3. // 镜像仓库项目
  4. def project = "jenkinsci"
  5. // 镜像名称
  6. def app_name = "citest"
  7. // 镜像完整名称
  8. def image_name = "${registry}/${project}/${app_name}:${BUILD_NUMBER}"
  9. // git仓库地址
  10. def git_address = "http://139.9.134.177:10080/miao/citest.git"
  12. // 认证
  13. def harbor_auth = "db4b7f06-7df6-4da7-b5b1-31e91b7a70e3"
  14. def gitlab_auth = "53d88c8f-3063-4048-9205-19fc6222b887"
  16. podTemplate(
  17.     label: 'jenkins-agent',
  18.     cloud: 'kubernetes',
  19.     containers: [
  20.         containerTemplate(
  21.             name: 'jnlp',
  22.             image: "${registry}/jenkinsci/jenkins-slave-jdk:1.8"
  23.         )
  24.     ],
  25.     volumes: [
  26.         hostPathVolume(mountPath: '/var/run/docker.sock', hostPath: '/var/run/docker.sock'),
  27.         hostPathVolume(mountPath: '/usr/bin/docker', hostPath: '/usr/bin/docker')
  28.     ]
  29. )
  30. {
  31.   node("jenkins-agent"){
  32.         stage('拉取代码') { // for display purposes
  33.             checkout([$class: 'GitSCM', branches: [[name: '${Branch}']], userRemoteConfigs: [[credentialsId: "${gitlab_auth}", url: "${git_address}"]]])
  34.             sh "ls"
  35.         }
  36.         stage('代码编译') {
  37.             sh "mvn clean package -Dmaven.test.skip=true"
  38.             sh "ls"
  39.         }
  40.         stage('构建镜像') {
  41.             withCredentials([usernamePassword(credentialsId: "${harbor_auth}", passwordVariable: 'password', usernameVariable: 'username')]) {
  42. 				sh """
  43. 					echo '
  44. 						FROM tomcat
  45. 						LABEL maintainer miaocunfa
  46. 						RUN rm -rf /usr/local/tomcat/webapps/*
  47. 						ADD target/*.war /usr/local/tomcat/webapps/ROOT.war
  48. 					' > Dockerfile
  50. 					docker build -t ${image_name} .
  51. 					docker login -u ${username} -p '${password}' ${registry}
  52. 					docker push ${image_name}
  53. 				"""
  54. 			}
  55. 		}
  56. 	}
  57. }

写脚本用来提交gitlab

  1. [root@support ~]# cat gitpush.sh
  2. testdate=$(date)
  3. cd /root/citest
  4. echo $testdate >> pod-slave.log
  5. git add -A
  6. git commit -m "$testdate"
  7. git push origin master

代码提交已经触发了编号为33的任务开始构建。

jenkins构建过程中的日志。

jenkins构建成功后，harbor中已经有了标签为33的镜像。

4.7、Jenkins在Kubernetes中持续部署

已经成功使用jenkins构建好镜后，接下来完成将镜像部署在K8s平台。这个过程我们需要用到插件Kubernetes Continuous Deploy Plugin

4.7.1、k8s认证

将.kube/config的内容拷贝至jenkins中生成凭据

拷贝凭据的id到pipeline脚本中引用

4.7.2、k8s添加harbor仓库secret

  1. [root@master ~]# kubectl create secret docker-registry harbor-pull-secret --docker-server='http://10.0.0.123:8080' --docker-username='admin' --docker-password='Harbor12345'
  2. secret/harbor-pull-secret created

4.7.3、pipeline脚本

  1. // 镜像仓库地址
  2. def registry = "10.0.0.123:8080"
  3. // 镜像仓库项目
  4. def project = "jenkinsci"
  5. // 镜像名称
  6. def app_name = "citest"
  7. // 镜像完整名称
  8. def image_name = "${registry}/${project}/${app_name}:${BUILD_NUMBER}"
  9. // git仓库地址
  10. def git_address = "http://139.9.134.177:10080/miao/citest.git"
  12. // 认证
  13. def harbor_auth = "db4b7f06-7df6-4da7-b5b1-31e91b7a70e3"
  14. def gitlab_auth = "53d88c8f-3063-4048-9205-19fc6222b887"
  16. // K8s认证
  17. def k8s_auth = "586308fb-3f92-432d-a7f7-c6d6036350dd"
  18. // harbor仓库secret_name
  19. def harbor_registry_secret = "harbor-pull-secret"
  20. // k8s部署后暴露的nodePort
  21. def nodePort = "30666"
  23. podTemplate(
  24.     label: 'jenkins-agent',
  25.     cloud: 'kubernetes',
  26.     containers: [
  27.         containerTemplate(
  28.             name: 'jnlp',
  29.             image: "${registry}/jenkinsci/jenkins-slave-jdk:1.8"
  30.         )
  31.     ],
  32.     volumes: [
  33.         hostPathVolume(mountPath: '/var/run/docker.sock', hostPath: '/var/run/docker.sock'),
  34.         hostPathVolume(mountPath: '/usr/bin/docker', hostPath: '/usr/bin/docker')
  35.     ]
  36. )
  37. {
  38.   node("jenkins-agent"){
  39.         stage('拉取代码') { // for display purposes
  40.             checkout([$class: 'GitSCM', branches: [[name: '${Branch}']], userRemoteConfigs: [[credentialsId: "${gitlab_auth}", url: "${git_address}"]]])
  41.             sh "ls"
  42.         }
  43.         stage('代码编译') {
  44.             sh "mvn clean package -Dmaven.test.skip=true"
  45.             sh "ls"
  46.         }
  47.         stage('构建镜像') {
  48.             withCredentials([usernamePassword(credentialsId: "${harbor_auth}", passwordVariable: 'password', usernameVariable: 'username')]) {
  49. 				sh """
  50. 					echo '
  51. 						FROM tomcat
  52. 						LABEL maintainer miaocunfa
  53. 						RUN rm -rf /usr/local/tomcat/webapps/*
  54. 						ADD target/*.war /usr/local/tomcat/webapps/ROOT.war
  55. 					' > Dockerfile
  57. 					docker build -t ${image_name} .
  58. 					docker login -u ${username} -p '${password}' ${registry}
  59. 					docker push ${image_name}
  60. 				"""
  61. 			}
  62. 		}
  63. 		stage('部署到K8s'){
  64.             sh """
  65.                 sed -i 's#\$IMAGE_NAME#${image_name}#' deploy.yml
  66.                 sed -i 's#\$SECRET_NAME#${harbor_registry_secret}#' deploy.yml
  67.                 sed -i 's#\$NODE_PORT#${nodePort}#' deploy.yml
  68.             """
  69.             kubernetesDeploy configs: 'deploy.yml', kubeconfigId: "${k8s_auth}"
  70. 		}
  71. 	}
  72. }
deploy.yaml

用来将镜像部署为deployment控制器控制的pod，放在代码仓库中跟代码一起推送。

  1. kind: Deployment
  2. apiVersion: apps/v1
  3. metadata:
  4.   name: web
  5. spec:
  6.   replicas: 3
  7.   selector:
  8.     matchLabels:
  9.       app: java-demo
  10.   template:
  11.     metadata:
  12.       labels:
  13.         app: java-demo
  14.     spec:
  15.       imagePullSecrets:
  16.       - name: $SECRET_NAME
  17.       containers:
  18.       - name: tomcat
  19.         image: $IMAGE_NAME
  20.         ports:
  21.         - containerPort: 8080
  22.           name: web
  23.         livenessProbe:
  24.           httpGet:
  25.             path: /
  26.             port: 8080
  27.           initialDelaySeconds: 20
  28.           timeoutSeconds: 5
  29.           failureThreshold: 3
  30.         readinessProbe:
  31.           httpGet:
  32.             path: /
  33.             port: 8080
  34.           initialDelaySeconds: 20
  35.           timeoutSeconds: 5
  36.           failureThreshold: 3
  38. ---
  40. kind: Service
  41. apiVersion: v1
  42. metadata:
  43.   name: web
  44. spec:
  45.   type: NodePort
  46.   selector:
  47.     app: java-demo
  48.   ports:
  49.     - protocol: TCP
  50.       port: 80
  51.       targetPort: 8080
  52.       nodePort: $NODE_PORT

4.7.4、推送

下面是整个完整的CI/CD流程

1、git推送代码至gitlab代码仓库

2、gitlab使用webhook触发jenkins任务

左下角webhook已经触发，编号为53的jenkins任务已经开始

jenkins任务流程

3、harbor镜像仓库

tag标签为53的镜像也已经推送至harbor

4、使用kubectl监控pods的变化

jenkins在任务流程中会先构建slave pod，在执行完将镜像部署到kubernetes后，slave pod会销毁，web镜像处于running状态。

5、邮件通知

在整个jenkins任务执行成功后，发送邮件通知

邮件的配置会在4.8优化部分贴出来。

4.8、优化部分

4.8.1、pipeline脚本跟代码一起托管

Jenkinsfile放在代码仓库的好处就是，可以对Jenkinsfile也做一个版本的管理，与当前项目生命周期是一致的。

首先将pipeline脚本保存至本地git仓库中，文件名为Jenkinsfile

jenkins配置如下

4.8.2、构建成功后添加邮件通知

1、邮件通知需要用到已经安装好的一个插件Email Extension

2、Email Extension的配置

3、邮件模板内容，html模板

4、系统默认邮件服务配置，配置完可以发送测试邮件。

5、测试邮件内容

邮件模板
  1. <!DOCTYPE html>
  2. <html>
  3. <head>
  4. <meta charset="UTF-8">
  5. <title>${ENV, var="JOB_NAME"}-第${BUILD_NUMBER}次构建日志</title>
  6. </head>    
  8. <body leftmargin="8" marginwidth="0" topmargin="8" marginheight="4"
  9.     offset="0">
  10.     <table width="95%" cellpadding="0" cellspacing="0"  style="font-size: 11pt; font-family: Tahoma, Arial, Helvetica, sans-serif">
  11.         <tr>
  12.             本邮件由系统自动发出，无需回复！<br/>
  13.             各位同事，大家好，以下为${PROJECT_NAME }项目构建信息</br>
  14.             <td><font color="#CC0000">构建结果 - ${BUILD_STATUS}</font></td>
  15.         </tr>
  16.         <tr>
  17.             <td><br />
  18.             <b><font color="#0B610B">构建信息</font></b>
  19.             <hr size="2" width="100%" align="center" /></td>
  20.         </tr>
  21.         <tr>
  22.             <td>
  23.                 <ul>
  24.                     <li>项目名称 ： ${PROJECT_NAME}</li>
  25.                     <li>构建编号 ： 第${BUILD_NUMBER}次构建</li>
  26.                     <li>触发原因 ： ${CAUSE}</li>
  27.                     <li>构建状态 ： ${BUILD_STATUS}</li>
  28. 		    <li>构建信息 ： <a href="${BUILD_URL}">${BUILD_URL}</a></li>
  29.                     <li>构建日志 ： <a href="${BUILD_URL}console">${BUILD_URL}console</a></li>
  30.                     <li>构建历史 ： <a href="${PROJECT_URL}">${PROJECT_URL}</a></li>
  31. 		    <!--<li>部署地址 ： <a href="${project_url}">${project_url}</a></li>-->
  32.                 </ul>    
  34. 				<h4><font color="#0B610B">失败用例</font></h4>
  35. 				<hr size="2" width="100%" />
  36. 				$FAILED_TESTS<br/>
  38. 				<h4><font color="#0B610B">最近提交(#$SVN_REVISION)</font></h4>
  39. 				<hr size="2" width="100%" />
  40. 				<ul>
  41. 				${CHANGES_SINCE_LAST_SUCCESS, reverse=true, format="%c", changesFormat="<li>%d [%a] %m</li>"}
  42. 				</ul>
  43. 				<font color="#0B610B">详细提交: </font><a href="${PROJECT_URL}changes">${PROJECT_URL}changes</a><br/>
  45.             </td>
  46.         </tr>
  47.     </table>
  48. </body>
  49. </html>

在持续集成这一块我还是一个初学者，期望得到您的指点。

构建gitlab+Jenkins+harbor+kubernetes的DevOps持续集成持续部署环境的更多相关文章

  1. GitLab + Jenkins + Harbor 工具链快速落地指南

    目录 一.今天想干啥? 二.今天干点啥? 三.今天怎么干? 3.1.常规打法 3.2.不走寻常路 四.开干吧! 4.1.工具链部署 4.2.网络配置 4.3.验证工具链部署结果 4.3.1.GitLa ...

  2. GitLab + Jenkins + Docker + Kubernetes。

    目前方案是GitLab + Jenkins + Docker + Kubernetes. 方案的工作流程如下:首先,开发人员提交代码代码提交:随后,GitLab 会自动触发Jenkins job,Je ...

  3. 03 持续集成和部署/基础设施 - DevOps之路

    02 持续集成和部署/基础设施 - DevOps之路 文章Github地址,欢迎start:https://github.com/li-keli/DevOps-WiKi 服务的持续集成和部署这里有两套 ...

  4. Linux-GitLab+Jenkins持续集成+自动化部署

    GitLab+Jenkins持续集成+自动化部署 什么是持续集成? (1)Continuous integration (CI) 持续集成是一种软件开发实践,即团队开发成员经常集成他们的工作,通常每个 ...

  5. 持续集成①安装部署jenkins从git获取代码

    持续集成①安装部署jenkins从git获取代码 一:持续集成的概念: 1.1:总体的概括 持续集成Continuous Integration 持续交付Continuous Delivery 持续部 ...

  6. Jenkins持续集成学习-Windows环境进行.Net开发4

    目录 Jenkins持续集成学习-Windows环境进行.Net开发4 目录 前言 目标 Github持续集成 提交代码到Github 从Github更新代码 git上显示构建状态 自动触发构建 Gi ...

  7. vsts + XX云服务器构建netcore+docker持续集成交付部署

    持续集成交付部署是什么意思,它给我们带来什么好处? 先贴一张图 持续集成(Continuous Integration) 持续集成强调开发人员提交了新代码之后,立刻进行构建.(单元)测试(这个要看情况 ...

  8. Jenkins持续集成学习-Windows环境进行.Net开发1

    目录 Jenkins持续集成学习-Windows环境进行.Net开发 目录 前言 目标 使用Jenkins 安装 添加.net环境配置 部署 结语 参考文档 Jenkins持续集成学习-Windows ...

  9. Jenkins持续集成学习-Windows环境进行.Net开发2

    目录 Jenkins持续集成学习-Windows环境进行.Net开发2 目录 前言 目标 使用 .Net Stardard 单元测试 流程 手动执行单元测试 自动执行单元测试 单元测试报告 上传Nug ...

随机推荐

  1. 编写一个函数来找出所有不带歧义的函数名，也就是 那些只在一个模块里出现过的函数名(erlang)

    erlang程序设计第八章练习题第二题: code:all_loaded()命令会返回一个由{Mod,File}对构成的列表,内含所有Erlang系统 载入的模块.使用内置函数Mod:module_i ...

  2. myslq5.7安装以及root密码找回

    一.mysql安装 创建用户和用户组: groupadd mysqluseradd -g mysql mysql -s /sbin/nologin 解压压缩文件并创建软链接 tar -xvf mysq ...

  3. 史无前例的RNN讲解

    这篇博客不是一篇讲解原理的博客,这篇博客主要讲解time_steps,如果这篇博客没有让你明白time_steps,那么算我无能. 我曾翻阅各大网站,各大博客,他们的对RNN中time_steps的讲 ...

  4. gallery的简单使用方法

    Gallery意思是"画廊",就是一个水平可左右拖动的列表,里面可以放置多个图片,经常和ImageSwitcher一起使用 在使用ImageSwitcher时,需要传入一个View ...

  5. CentOS7上OpenResty安装

    1,OpenResty安装 通过repl源安装: sudo yum-config-manager --add-repo https://openresty.org/yum/cn/centos/Open ...

  6. 01 (OC)* @property 后面可以有哪些修饰符？

    一:@property 后面可以有哪些修饰符? 1:线程安全的: atomic,nonatomic 2:访问权限的 readonly,readwrite 3:内存管理(ARC) assign, cop ...

  7. Spring事务传播行为中可能的坑点

    一.简介 Spring事务配置及相关说明详见:https://www.cnblogs.com/eric-fang/p/11052304.html.这里说明spring事务的几点注意: 1.默认只会检查 ...

  8. 判断dom原始是否在可视区域内

    isElementInViewport (el, offset = 0) { const box = el.getBoundingClientRect(), top = (box.top >= ...

  9. Unity-遇到的问题小总结

    1. event trigger后面显示不了对应的方法 原因:我是直接把脚本拖拽进去的,这是没有实例化的,拖拽进去的应该是挂载这个脚本的GameObject就可以了 2.制作prefeb 将场景中的单 ...

  10. 『嗨威说』算法设计与分析 - STL中Sort函数的实现原理初探

    本文索引目录: 一.对Sort算法实现的个人阅读体会 二.Sort算法使用的三个排序算法的优点介绍 2.1 插入排序的优缺点 2.2 堆排序的优缺点 2.3 快速排序的优缺点 2.4 新的结合排序—— ...