云原生之旅 - 11)基于 Kubernetes 动态伸缩 Jenkins Build Agents
前言
上一篇文章 云原生之旅 - 10)手把手教你安装 Jenkins on Kubernetes 我们介绍了在 Kubernetes 上安装 Jenkins,本文介绍下如何设置k8s pod作为Jenkins 构建job的 agent。
Jenkins master 和 agent 均以 pod 的形式运行在 Kubernetes 节点上。Master 运行在其中一个节点上,其配置数据 Jenkins home 使用存储卷挂载,master pod重启不会导致数据丢失。agent 运行在各个节点上,根据需求动态创建并自动释放。这样做的好处很多,比如高可用,高伸缩性,资源利用率高。
关键词:Jenkins on Kubernetes 实践,Jenkins 和 Kubernetes,在Kubernetes上安装Jenkins,Jenkins 高可用安装,Jenkins 动态伸缩构建, Kubernetes Pod as Jenkins build agent
准备
- 已搭建 Jenkins master on kubernetes 云原生之旅 - 10)手把手教你安装 Jenkins on Kubernetes
- 准备一个 Service Account,对目标 cluster 具有k8s admin权限,以便部署。
- 防火墙已开通 Jenkins 出站到Docker hub,方便 push/pull image
- 防火墙已开通 Jenkins 到 目标 cluster,以便部署。
插件安装
- Kubernetes Plugin
- Google Kubernetes Engine Plugin (我的例子是部署到 GKE cluster)
Jenkins 配置
Manage Nodes and Clouds
10. Click `Add Pod Template` then `Pod Template Details`
Ensure that you remove the sleep and 9999999 default argument from the container template.
Manage Credentials
- Add `Usernames with password` for docker hub account/pwd,比如 wade_test_dockerhub
- Add `Google Service Account from private key` 比如 gcp_sa_json_key
Credentials 会在Jenkinsfile里面用到。
### 本文首发于博客园 https://www.cnblogs.com/wade-xu/p/16863955.html
Test a freestyle project
这个label 和我们上面创建 pod template时用的label一致. 这样的话 Jenkins就知道用哪个 pod template 作为 agent container.
点Build Now
Agent jenkins-agent-l7hw9 is provisioned from template jenkins-agent ...... Building remotely on jenkins-agent-l7hw9 (kubeagent) in workspace /home/jenkins/agent/workspace/quick-test
[quick-test] $ /bin/sh -xe /tmp/jenkins17573873264046707236.sh
+ echo test pipeline
test pipeline
Finished: SUCCESS
### 本文首发于博客园 https://www.cnblogs.com/wade-xu/p/16863955.html
Jenkinsfile
CI
kind: Pod
spec:
containers: # list of containers that you want present for your build, you can define a default container in the Jenkinsfile
- name: maven
image: maven:3.5.4-jdk-8-slim
command: ["tail", "-f", "/dev/null"] # this or any command that is bascially a noop is required, this is so that you don't overwrite the entrypoint of the base container
imagePullPolicy: Always # use cache or pull image for agent
resources: # request and limit the resources your build contaienr
requests:
memory: 4Gi
cpu: 2
limits:
memory: 4Gi
cpu: 2
volumeMounts:
- mountPath: /root/.m2 # maven .m2 cache directory
name: maven-home
- name: git
image: bitnami/git:2.38.1
imagePullPolicy: IfNotPresent
command: ["tail", "-f", "/dev/null"]
resources: # limit the resources your build contaienr
limits:
cpu: 100m
memory: 256Mi
- name: kubectl-kustomize
image: line/kubectl-kustomize:1.25.3-4.5.7
imagePullPolicy: IfNotPresent
command: ["tail", "-f", "/dev/null"]
resources: # limit the resources your build contaienr
limits:
cpu: 100m
memory: 256Mi
- name: docker
image: docker:18.06.1
command: ["tail", "-f", "/dev/null"]
imagePullPolicy: Always
volumeMounts:
- name: docker
mountPath: /var/run/docker.sock # We use the k8s host docker engine
volumes:
- name: docker
hostPath:
path: /var/run/docker.sock
- name: maven-home
persistentVolumeClaim:
claimName: maven-repo-storage
build-pod.yaml
在Jenkinsfile里面定义agent 使用这个yaml file
agent {
kubernetes {
idleMinutes 3 // how long the pod will live after no jobs have run on it
yamlFile './build-pod.yaml' // path to the pod definition relative to the root of our project
defaultContainer 'docker' // define a default container if more than a few stages use it, otherwise default to jnlp container
}
下面步骤是 docker login/build/tag/push
stage('Build and Push Docker Image') {
steps {
script {
dir(dir_path) {
container('docker') {
// docker login, Using single-quotes instead of double-quotes when referencing these sensitive environment variables prevents this type of leaking.
sh 'echo $DOCKER_HUB_CREDS_PSW | docker login -u $DOCKER_HUB_CREDS_USR --password-stdin $DOCKER_HUB_REGISTRY'
// build image with git tag
sh """
docker build -t $PROJECT_IMAGE_WITH_TAG .
docker tag $PROJECT_IMAGE_WITH_TAG $DOCKER_HUB_CREDS_USR/$PROJECT_IMAGE_WITH_TAG
""" // push image_tag to docker hub
sh """
docker push $DOCKER_HUB_CREDS_USR/$PROJECT_IMAGE_WITH_TAG
"""
}
}
}
}
}
我这里没有选择用 docker.withRegistry
docker.withRegistry("$DOCKER_HUB_REGISTRY", "$DOCKER_HUB_CREDENTIAL") {}
因为会有不安全的log提示
WARNING! Using --password via the CLI is insecure. Use --password-stdin.
CI + Kustomize + CD
这个例子是上面的 CI 之后 加上 - 利用 Kustomize build K8S resource manifests 然后 CD 到一个 Cluster
Kustomize 可以参考 云原生之旅 - 6)不能错过的一款 Kubernetes 应用编排管理神器 Kustomize
// assume your k8s manifests in another repo, mine is same repo, just in order to show git clone step
stage('Checkout K8S manifests') {
steps {
script {
dir(dir_path) {
container('git') {
if (! fileExists('learning_by_doing/README.md')) {
sh """
git clone https://github.com/wadexu007/learning_by_doing.git
ls -lhrt
"""
} else {
sh 'echo manifes repo already exist.'
}
}
}
}
}
} stage('Build manifests with Kustomize') {
steps {
script {
dir(dir_path) {
container('kubectl-kustomize') {
sh """
cd learning_by_doing/Kustomize/demo-manifests/services/demo-app/dev/
kustomize edit set image $DOCKER_HUB_CREDS_USR/$PROJECT_IMAGE_WITH_TAG
kustomize build > $WORKSPACE/$dir_path/deployment.yaml
"""
}
}
}
}
} stage('Deploy to GKE test cluster') {
environment{
PROJECT_ID = 'xperiences-eng-cn-dev'
CLUSTER_NAME = 'xpe-spark-test-gke'
REGION = 'asia-east2'
CREDENTIALS_ID = 'gcp_sa_json_key'
}
steps {
script {
dir(dir_path) {
container('kubectl-kustomize') {
sh """
chown 1000:1000 deployment.yaml
echo start to deploy to cluster $CLUSTER_NAME
"""
step([
$class: 'KubernetesEngineBuilder',
projectId: env.PROJECT_ID,
clusterName: env.CLUSTER_NAME,
location: env.REGION,
manifestPattern: 'deployment.yaml',
credentialsId: env.CREDENTIALS_ID,
verifyDeployments: false])
// verifyDeployments does not work for non-default namespace
}
}
}
}
}
Pipeline: Input Step
stage('Wait for SRE Approval') {
steps {
timeout(time:72, unit:'HOURS') {
input message: "Approved Prod deployment?", submitter: 'sre-team'
}
}
} // deployment to multipe k8s clusters
stage('Deploy to GKE Prod cluster') {
environment{
PROJECT_ID = 'sre-cn-dev'
CREDENTIALS_ID = 'gcp_sa_json_key'
CLUSTER_COMMON_NAME = 'demo-gke-prod'
}
steps {
script {
env.REGION = input message: 'Choose which region you want to deploy?',
parameters: [choice(name: 'Region',
description: 'Select Region to Deloy',
choices: ['europe-west1', 'us-central1'])
]
dir(dir_path) {
if ( env.REGION == "europe-west1" ) {
def eu_cluster_name = env.CLUSTER_COMMON_NAME + "-eu"
container('kubectl-kustomize') {
sh "echo deploy to cluster $eu_cluster_name in region: $REGION"
}
}
if ( env.REGION == "us-central1" ) {
def us_cluster_name = env.CLUSTER_COMMON_NAME + "-us"
container('kubectl-kustomize') {
sh "echo deploy to cluster $us_cluster_name in region: $REGION"
}
}
}
}
}
}
所有例子均在我的 github repo。
### 本文首发于博客园 https://www.cnblogs.com/wade-xu/p/16863955.html
测试
总结
最佳实践
- 设置 resource requests and limits on each container in your Pod
- 如果使用maven 构建 java项目,.m2 cache目录需要 mount 出来,这样加快后面的maven build速度。
- 使用 Jenkins Shared Libraries 抽取Pipeline的共用代码
- 在容器里构建容器化应用(Run docker in docker) 我的例子是通过 mount docker.sock 利用k8s 主机 docker engine来实现的,这种方式需要 privileges mode 不安全,推荐使用Kaniko,下一篇文章会介绍。
云原生之旅 - 11)基于 Kubernetes 动态伸缩 Jenkins Build Agents的更多相关文章
- 云原生之旅 - 5)Kubernetes时代的包管理工具 Helm
前言 上一篇文章 [基础设施即代码 使用 Terraform 创建 Kubernetes] 教会了你如何在Cloud上面建Kubernetes资源,那么本篇来讲一下如何在Kubernetes上面部署应 ...
- 云原生之旅 - 6)不能错过的一款 Kubernetes 应用编排管理神器 Kustomize
前言 相信经过前一篇文章的学习,大家已经对Helm有所了解,本篇文章介绍另一款工具 Kustomize,为什么Helm如此流行,还会出现 Kustomize?而且 Kustomize 自 kubect ...
- 云原生之旅 - 10)手把手教你安装 Jenkins on Kubernetes
前言 谈到持续集成工具就离不开众所周知的Jenkins,本文带你了解如何在 Kubernetes 上安装 Jenkins,后续文章会带你深入了解如何使用k8s pod 作为 Jenkins的build ...
- 云原生之旅 - 8)云原生时代的网关 Ingress Nginx
前言 当我们在Kubernetes部署的服务需要暴露给外部用户使用时,有三种选择:LoadBalancer,NodePort, Ingress. LoadBalancer类型得结合各个Cloud Pr ...
- 云原生之旅 - 7)部署Terrform基础设施代码的自动化利器 Atlantis
前言 前面有几篇文章讲述了如何使用Terraform创建资源 (基础设施即代码 Terraform 快速入门, 使用 Terraform 创建 Kubernetes) 以及 Kubernetes时代的 ...
- 基于Kubernetes/K8S构建Jenkins持续集成平台(上)-2
基于Kubernetes/K8S构建Jenkins持续集成平台(上)-2 Kubernetes实现Master-Slave分布式构建方案 传统Jenkins的Master-Slave方案的缺陷 Mas ...
- 基于Kubernetes/K8S构建Jenkins持续集成平台(下)
基于Kubernetes/K8S构建Jenkins持续集成平台(下) Jenkins-Master-Slave架构图回顾: 安装和配置NFS NFS简介 NFS(Network File System ...
- 基于Kubernetes/K8S构建Jenkins持续集成平台(上)-1
基于Kubernetes/K8S构建Jenkins持续集成平台(上)-1 Jenkins的Master-Slave分布式构建 什么是Master-Slave分布式构建 Jenkins的Master-S ...
- 云原生之旅 - 9)云原生时代网关的后起之秀Envoy Proxy 和基于Envoy 的 Emissary Ingress
前言 前一篇文章讲述了基于Nginx代理的Kuberenetes Ingress Nginx[云原生时代的网关 Ingress Nginx]这次给大家介绍下基于Envoy的 Emissary Ingr ...
随机推荐
- Maven中使用ssm框架出现:org.apache.tomcat.util.modeler.BaseModelMBean.invoke 调用方法[manageApp]时发生异常
org.apache.tomcat.util.modeler.BaseModelMBean.invoke 调用方法[manageApp]时发生异常 首先可以排查一下像: @RequestMapping ...
- C#使用Spire.Pdf包对PDF文档进行数字签名
背景 对PDF文档进行数字签名的需求 对PDF文档添加水印的需求 网上资料版本不一或不全 本文章提到的Spire.Pdf均是使用的Spire.Pdf for .NET,除此之前还有其他语言的版本,如S ...
- 搭建docker镜像仓库(二):使用harbor搭建本地镜像仓库
目录 一.系统环境 二.前言 三.Harbor 四.使用harbor搭建私有镜像仓库 4.1 环境介绍 4.2 k8smaster节点安装配置harbor 4.2.1 安装harbor离线包 4.2. ...
- Spring配置XML本地提示
Spring配置XML本地提示:点击eclipse属性-->选择XML Catalog 这里有一点要注意:要选择schema location
- 3.云原生之Docker容器三大核心概念介绍
转载自:https://www.bilibili.com/read/cv15181760/?from=readlist docker search --no-trunc=false [镜像名称] #搜 ...
- Elasticsearch 数据建模指南
文章转载自:https://mp.weixin.qq.com/s/vSh6w3eL_oQvU1mxnxsArA 0.题记 我在做 Elasticsearch 相关咨询和培训过程中,发现大家普遍更关注实 ...
- kubeadm init 命令执行流程
- 为MinIO Server设置Nginx代理
官方文档地址:http://docs.minio.org.cn/docs/master/setup-nginx-proxy-with-minio nginx参考网址:https://www.nginx ...
- Traefik开启监控,日志,追踪需要的参数
监控 官方文档地址:https://doc.traefik.io/traefik/observability/metrics/overview/ 可以使用多种监控软件,比如Datadog,Influx ...
- Centos7下的基本操作
本系统是在centos7下最小化安装的 文件操作相关 创建文件夹mkdir name //创建一个文件夹 创建文本touch test.txt //创建文本 删除文件夹rm -rf 文件名 //删除文 ...