使用kubeadm部署kubernetes1.9.1+coredns+kube-router(ipvs)高可用集群
由于之前已经写了两篇部署kubernetes的文章，整个过程基本一致，所以这篇只着重说一下coredns和kube-router的部署。
kube version: 1.9.1
docker version: 17.03.2-ce
OS version: debian stretch
依然是三个master节点、一个node节点。
1、准备镜像，自行科学下载。
# docker images| grep 1.9.1
gcr.io/google_containers/kube-apiserver-amd64 v1.9.1
gcr.io/google_containers/kube-controller-manager-amd64 v1.9.1
gcr.io/google_containers/kube-scheduler-amd64 v1.9.1
gcr.io/google_containers/kube-proxy-amd64 v1.9.1
2、安装新版本的kubeadm、kubectl、kubelet。
# aptitude install -y kubeadm kubectl kubelet
3、部署第一个master节点。准备kubeadm的配置文件，这里官方提供的配置说明并不完善，应该说无法使用，经过了一番查找和测试。
# cat kubeadm-config-191.yml
apiVersion: kubeadm.k8s.io/v1alpha1
kind: MasterConfiguration
api:
advertiseAddress: "192.168.5.62"
etcd:
endpoints:
- "http://192.168.5.84:2379"
- "http://192.168.5.85:2379"
- "http://192.168.2.77:2379"
kubernetesVersion: "v1.9.1"
apiServerCertSANs:
- uy06-04
- uy06-05
- uy08-10
- uy08-11
- 192.168.6.16
- 192.168.6.17
- 127.0.0.1
- 192.168.5.62
- 192.168.5.63
- 192.168.5.107
- 192.168.5.108
- 30.0.0.1
- 10.96.0.1
- kubernetes
- kubernetes.default
- kubernetes.default.svc
- kubernetes.default.svc.cluster
- kubernetes.default.svc.cluster.local
tokenTTL: 0s
networking:
podSubnet: 30.0.0.0/10
apiServerExtraArgs:
enable-swagger-ui: "true"
insecure-bind-address: 0.0.0.0
insecure-port: "8088"
endpoint-reconciler-type: "lease"
controllerManagerExtraArgs:
address: 0.0.0.0
schedulerExtraArgs:
address: 0.0.0.0
featureGates:
CoreDNS: true
kubeProxy:
config:
featureGates: "SupportIPVSProxyMode=true"
mode: "ipvs"
需要提醒一下的是，这里开启的是kube-proxy的ipvs模式，部署的时候部署的依然是kube-proxy，而不是kube-router。
如果你打算使用kube-router作为网络插件，是可以不考虑kube-proxy的配置的，kube-proxy会被删掉。kube-router不仅取代kube-proxy代理svc，并且它也是网络插件。
4、使用kubeadm执行初始化。
# kubeadm init --config=kubeadm-config-191.yml --ignore-preflight-errors=all
[init] Using Kubernetes version: v1.9.1
[init] Using Authorization modes: [Node RBAC]
[preflight] Running pre-flight checks.
[WARNING CRI]: unable to check if the container runtime at "/var/run/dockershim.sock" is running: exit status 1
[certificates] Generated ca certificate and key.
[certificates] Generated apiserver certificate and key.
[certificates] apiserver serving cert is signed for DNS names [uy06-04 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local uy06-04 uy06-05 uy08-10 uy08-11 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 192.168.5.62 192.168.6.16 192.168.6.17 127.0.0.1 192.168.5.62 192.168.5.63 192.168.5.107 192.168.5.108 30.0.0.1 10.96.0.1]
[certificates] Generated apiserver-kubelet-client certificate and key.
[certificates] Generated sa key and public key.
[certificates] Generated front-proxy-ca certificate and key.
[certificates] Generated front-proxy-client certificate and key.
[certificates] Valid certificates and keys now exist in "/etc/kubernetes/pki"
[kubeconfig] Wrote KubeConfig file to disk: "admin.conf"
[kubeconfig] Wrote KubeConfig file to disk: "kubelet.conf"
[kubeconfig] Wrote KubeConfig file to disk: "controller-manager.conf"
[kubeconfig] Wrote KubeConfig file to disk: "scheduler.conf"
[controlplane] Wrote Static Pod manifest for component kube-apiserver to "/etc/kubernetes/manifests/kube-apiserver.yaml"
[controlplane] Wrote Static Pod manifest for component kube-controller-manager to "/etc/kubernetes/manifests/kube-controller-manager.yaml"
[controlplane] Wrote Static Pod manifest for component kube-scheduler to "/etc/kubernetes/manifests/kube-scheduler.yaml"
[init] Waiting for the kubelet to boot up the control plane as Static Pods from directory "/etc/kubernetes/manifests".
[init] This might take a minute or longer if the control plane images have to be pulled.
[apiclient] All control plane components are healthy after 90.501851 seconds
[uploadconfig] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[markmaster] Will mark node uy06-04 as master by adding a label and a taint
[markmaster] Master uy06-04 tainted and labelled with key/value: node-role.kubernetes.io/master=""
[bootstraptoken] Using token: b1bd11.9ecfaaad5274f9d1
[bootstraptoken] Configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstraptoken] Configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstraptoken] Configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstraptoken] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy
Your Kubernetes master has initialized successfully!
To start using your cluster, you need to run the following as a regular user:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
https://kubernetes.io/docs/concepts/cluster-administration/addons/
You can now join any number of machines by running the following on each node
as root:
kubeadm join --token b1bd11.9ecfaaad5274f9d1 192.168.5.62:6443 --discovery-token-ca-cert-hash sha256:09438d4384c393880a5ac18e2d3d06b547dae7242061c18c03f0fbb1bad76ade
验证kube-proxy的mode：
# kubectl exec -it kube-proxy-hr48q -n kube-system -- sh
# ps -ef
UID PID PPID C STIME TTY TIME CMD
root 1 0 0 Jan11 ? 00:04:12 /usr/local/bin/kube-proxy --config=/var/lib/kube-proxy/config.conf
root 29012 0 0 19:24 ? 00:00:00 sh
root 29043 29012 0 19:24 ? 00:00:00 ps -ef
# cat /var/lib/kube-proxy/config.conf
apiVersion: kubeproxy.config.k8s.io/v1alpha1
bindAddress: 0.0.0.0
clientConnection:
acceptContentTypes: ""
burst: 10
contentType: application/vnd.kubernetes.protobuf
kubeconfig: /var/lib/kube-proxy/kubeconfig.conf
qps: 5
clusterCIDR: 30.0.0.0/10
configSyncPeriod: 15m0s
conntrack:
max: null
maxPerCore: 32768
min: 131072
tcpCloseWaitTimeout: 1h0m0s
tcpEstablishedTimeout: 24h0m0s
enableProfiling: false
featureGates: SupportIPVSProxyMode=true
healthzBindAddress: 0.0.0.0:10256
hostnameOverride: ""
iptables:
masqueradeAll: false
masqueradeBit: 14
minSyncPeriod: 0s
syncPeriod: 30s
ipvs:
minSyncPeriod: 0s
scheduler: ""
syncPeriod: 30s
kind: KubeProxyConfiguration
metricsBindAddress: 127.0.0.1:10249
mode: ipvs <- 这里
oomScoreAdj: -999
portRange: ""
resourceContainer: /kube-proxy
udpTimeoutMilliseconds: 250ms
5、使master节点参与调度。
# kubectl taint nodes --all node-role.kubernetes.io/master-
6、部署kube-router。
a、下载yaml文件。
# curl -L -O https://raw.githubusercontent.com/cloudnativelabs/kube-router/master/daemonset/kubeadm-kuberouter-all-features.yaml
b、修改busybox的镜像拉取策略为imagePullPolicy: IfNotPresent，因为这里自动拉取镜像时总是连接超时，导致pod无法启动。
c、应用yaml文件。
# kubectl apply -f kubeadm-kuberouter-all-features.yaml
7、这时，核心组件应该都运行起来了。
# kubectl get po --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system coredns-65dcdb4cf-mlr9j 1/1 Running 0 22h
kube-system kube-apiserver-uy06-04 1/1 Running 0 22h
kube-system kube-controller-manager-uy06-04 1/1 Running 0 22h
kube-system kube-proxy-hr48q 1/1 Running 0 22h
kube-system kube-router-9lh8x 1/1 Running 0 22h
kube-system kube-scheduler-uy06-04 1/1 Running 0 22h
8、删除kube-proxy，并清除iptables规则。
# kubectl delete ds kube-proxy -n kube-system
# docker run --privileged --net=host gcr.io/google_containers/kube-proxy-amd64:v1.7.3 kube-proxy --cleanup-iptables
9、部署另外两个master节点，尝试通过vip请求apiserver将node节点添加到集群，以及剩余的其他配置，请参考之前的两篇。
10、部署完成后是这个样子。
# kubectl get no
NAME STATUS ROLES AGE VERSION
uy02-07 Ready <none> 1d v1.9.1
uy05-13 Ready master 2d v1.9.1
uy08-07 Ready <none> 1d v1.9.1
uy08-08 Ready <none> 1d v1.9.1
# kubectl get po --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
default frontend-66d686db4b-jkbdk 1/1 Running 0 1d
default redis-master-5fd44c4c6-gf4zm 1/1 Running 0 1d
default redis-slave-74fc6595b4-kp8sl 1/1 Running 0 1d
default redis-slave-74fc6595b4-shtx6 1/1 Running 0 1d
default snowflake-5c98868c55-8crlt 1/1 Running 0 3h
default snowflake-5c98868c55-9psss 1/1 Running 0 1d
default snowflake-5c98868c55-ccsfc 1/1 Running 0 1d
default snowflake-5c98868c55-p2tjh 1/1 Running 0 1d
kube-system coredns-65dcdb4cf-bv95f 1/1 Running 0 2d
kube-system coredns-65dcdb4cf-cv48z 1/1 Running 0 1h
kube-system coredns-65dcdb4cf-grxkw 1/1 Running 0 1d
kube-system coredns-65dcdb4cf-n5kkm 1/1 Running 0 1d
kube-system heapster-7bddb97655-5hbsp 1/1 Running 0 1d
kube-system heapster-7bddb97655-8dqgd 1/1 Running 0 1h
kube-system heapster-7bddb97655-fd4mb 1/1 Running 0 1d
kube-system heapster-7bddb97655-gznsm 1/1 Running 0 1d
kube-system kube-apiserver-uy05-13 1/1 Running 0 1d
kube-system kube-apiserver-uy08-07 1/1 Running 0 1d
kube-system kube-apiserver-uy08-08 1/1 Running 0 1d
kube-system kube-controller-manager-uy05-13 1/1 Running 0 23h
kube-system kube-controller-manager-uy08-07 1/1 Running 0 23h
kube-system kube-controller-manager-uy08-08 1/1 Running 0 23h
kube-system kube-router-57mws 1/1 Running 0 1d
kube-system kube-router-j6rks 1/1 Running 0 2d
kube-system kube-router-mfwqv 1/1 Running 0 1d
kube-system kube-router-txp8p 1/1 Running 0 1d
kube-system kube-scheduler-uy05-13 1/1 Running 0 23h
kube-system kube-scheduler-uy08-07 1/1 Running 0 23h
kube-system kube-scheduler-uy08-08 1/1 Running 1 23h
kube-system kubernetes-dashboard-79cb6d66b9-74cf4 1/1 Running 0 3h
kubernator kubernator-659cf655b6-9prx2 1/1 Running 0 1d
monitoring alertmanager-main-0 2/2 Running 0 1h
monitoring alertmanager-main-1 2/2 Running 0 1h
monitoring alertmanager-main-2 2/2 Running 0 1h
monitoring grafana-6b67b479d5-zj66c 2/2 Running 0 1h
monitoring kube-state-metrics-6f7b5c94f-v42tj 2/2 Running 0 1h
monitoring node-exporter-5b8p2 1/1 Running 0 1h
monitoring node-exporter-m85xx 1/1 Running 0 1h
monitoring node-exporter-pg2qz 1/1 Running 0 1h
monitoring node-exporter-x9lb6 1/1 Running 0 1h
monitoring prometheus-k8s-0 2/2 Running 0 1h
monitoring prometheus-k8s-1 2/2 Running 0 1h
monitoring prometheus-operator-8697c7fff9-dpn9r 1/1 Running 0 1h
# kubectl get cs
NAME STATUS MESSAGE ERROR
scheduler Healthy ok
controller-manager Healthy ok
etcd-2 Healthy {"health": "true"}
etcd-1 Healthy {"health": "true"}
etcd-0 Healthy {"health": "true"}
# kubectl cluster-info
Kubernetes master is running at https://192.168.6.15:6443
Heapster is running at https://192.168.6.15:6443/api/v1/namespaces/kube-system/services/heapster/proxy
KubeDNS is running at https://192.168.6.15:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
补充：
安装ipvsadm查看lvs规则。
# aptitude install -y ipvsadm
# ipvsadm -ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 192.168.5.42:30001 rr
-> 20.0.2.17:9090 Masq 1 0 497
TCP 192.168.5.42:30211 rr
-> 20.0.2.12:80 Masq 1 0 497
TCP 192.168.5.42:30900 rr
-> 20.0.2.20:9090 Masq 1 0 250
-> 20.0.4.15:9090 Masq 1 0 250
TCP 192.168.5.42:30902 rr
-> 20.0.2.19:3000 Masq 1 0 500
TCP 192.168.5.42:30903 rr
-> 20.0.0.19:9093 Masq 1 0 166
-> 20.0.2.21:9093 Masq 1 0 166
-> 20.0.4.14:9093 Masq 1 0 166
TCP 192.168.5.42:31001 rr
-> 20.0.0.8:80 Masq 1 0 497
TCP 10.96.0.1:443 rr persistent 10800
-> 192.168.5.42:6443 Masq 1 2 0
-> 192.168.5.104:6443 Masq 1 0 0
-> 192.168.5.105:6443 Masq 1 0 0
TCP 10.96.0.10:53 rr
-> 20.0.0.2:53 Masq 1 0 0
-> 20.0.1.2:53 Masq 1 0 0
-> 20.0.2.2:53 Masq 1 0 0
-> 20.0.4.9:53 Masq 1 0 0
TCP 10.97.245.128:6379 rr
-> 20.0.0.7:6379 Masq 1 0 0
TCP 10.98.159.23:80 rr
-> 20.0.2.17:9090 Masq 1 0 0
TCP 10.101.179.96:8080 rr
-> 20.0.4.12:8080 Masq 1 0 0
TCP 10.101.209.232:80 rr
-> 20.0.2.12:80 Masq 1 0 0
TCP 10.101.255.18:9090 rr
-> 20.0.2.20:9090 Masq 1 0 0
-> 20.0.4.15:9090 Masq 1 0 0
TCP 10.104.53.117:8080 rr
-> 20.0.4.13:8080 Masq 1 0 0
TCP 10.105.5.201:3000 rr
-> 20.0.2.19:3000 Masq 1 0 0
TCP 10.105.21.201:80 rr
-> 20.0.0.4:8082 Masq 1 0 0
-> 20.0.1.3:8082 Masq 1 0 0
-> 20.0.2.3:8082 Masq 1 0 0
-> 20.0.4.10:8082 Masq 1 0 0
TCP 10.105.113.2:6379 rr
-> 20.0.1.8:6379 Masq 1 0 0
-> 20.0.2.8:6379 Masq 1 0 0
TCP 10.105.159.162:9093 rr
-> 20.0.0.19:9093 Masq 1 0 0
-> 20.0.2.21:9093 Masq 1 0 0
-> 20.0.4.14:9093 Masq 1 0 0
TCP 10.110.48.172:80 rr
-> 20.0.0.8:80 Masq 1 0 0
UDP 10.96.0.10:53 rr
-> 20.0.0.2:53 Masq 1 0 14
-> 20.0.1.2:53 Masq 1 0 15
-> 20.0.2.2:53 Masq 1 0 15
-> 20.0.4.9:53 Masq 1 0 15
使用kubeadm部署kubernetes1.9.1+coredns+kube-router(ipvs)高可用集群的更多相关文章
- Kubeadm 1.9 HA 高可用集群本地离线镜像部署【已验证】
k8s介绍 k8s 发展速度很快,目前很多大的公司容器集群都基于该项目,如京东,腾讯,滴滴,瓜子二手车,易宝支付,北森等等. kubernetes1.9版本发布2017年12月15日,每三个月一个迭代 ...
- [K8s 1.9实践]Kubeadm 1.9 HA 高可用 集群 本地离线镜像部署
k8s介绍 k8s 发展速度很快,目前很多大的公司容器集群都基于该项目,如京东,腾讯,滴滴,瓜子二手车,北森等等. kubernetes1.9版本发布2017年12月15日,每是那三个月一个迭代, W ...
- kubeadm部署高可用集群Kubernetes 1.14.1版本
Kubernetes高可用集群部署 部署架构: Master 组件: kube-apiserver Kubernetes API,集群的统一入口,各组件协调者,以HTTP API提供接口服务,所有对象 ...
- kubeadm使用外部etcd部署kubernetes v1.17.3 高可用集群
文章转载自:https://mp.weixin.qq.com/s?__biz=MzI1MDgwNzQ1MQ==&mid=2247483891&idx=1&sn=17dcd7cd ...
- kubeadm 使用 Calico CNI 以及外部 etcd 部署 kubernetes v1.23.1 高可用集群
文章转载自:https://mp.weixin.qq.com/s/2sWHt6SeCf7GGam0LJEkkA 一.环境准备 使用服务器 Centos 8.4 镜像,默认操作系统版本 4.18.0-3 ...
- kubeadm部署k8s1.9高可用集群--4部署master节点
部署master节点 kubernetes master 节点包含的组件: kube-apiserver kube-scheduler kube-controller-manager 本文档介绍部署一 ...
- [转帖]Breeze部署kubernetes1.13.2高可用集群
Breeze部署kubernetes1.13.2高可用集群 2019年07月23日 10:51:41 willblog 阅读数 673 标签: kubernetes 更多 个人分类: kubernet ...
- kubernetes kubeadm部署高可用集群
k8s kubeadm部署高可用集群 kubeadm是官方推出的部署工具,旨在降低kubernetes使用门槛与提高集群部署的便捷性. 同时越来越多的官方文档,围绕kubernetes容器化部署为环境 ...
- 使用二进制的方式部署 K8S-1.16 高可用集群
一.项目介绍 项目致力于让有意向使用原生kubernetes集群的企业或个人,可以方便的.系统的使用二进制的方式手工搭建kubernetes高可用集群.并且让相关的人员可以更好的理解kubernete ...
随机推荐
- 利用matplotlib的plot函数实现图像绘制
模式识别的一个实验,要求画出贝叶斯决策的图.这里我是利用python中的matplotlib库实现的图线的拟合.主要对于matplotlib的使用可以参照博客:webary 如果要绘制三维图像可以参考 ...
- windows下docker启动.net core mvc随手记
docker基本命令: 查看当前的版本docker--version查看本地所有镜像:docker images查看当前正在运行的所有容器docker ps停止某个容器:docker stop 容器I ...
- Linq 之 Select 和 where 的用法
最近开始学习linq.自己也总结一下,方便以后查阅. Select 同 Sql 中的 select 类似,即输出我们要的东东,感觉在 linq 中更加强大. Linq 可以对集合如数组.泛型等操作,这 ...
- M1/M2总结
在团队项目中,我是dev之一,负责的部分主要是排序部分. 应该说在团队中大家都各司其职,尤其是PM在组织方面也费了很多心思. 当然我个人因为空间上的距离和团队的联系不是那么密切…… 如同老师所言,这是 ...
- Linux课题实践一
Linux课题实践一 20135318 刘浩晨 1.1应用安装 (1)掌握软件源的维护方法,配置系统使用软件源镜像 删除过期或者重复的软件包:进入”系统设置“-”软件和更新”-”ubuntu软件“- ...
- Linux环境C程序设计
Linux基础 常用shell命令 命令 说明 命令 说明 man 查看联机帮助 ls 查看目录及文件列表 cp 复制目录或文件 mv 移动目录或文件 cd 改变文件或目录 rm 删除文件或目录 mk ...
- MySQLi面向对象实践--insert、update、delete
执行insert <?php $mysqli = new Mysqli(); $mysqli->connect("localhost","root" ...
- Golang 入门~~基础知识
变量声明 //通用形式,指定变量名,变量类型,变量值 var name int = 99 fmt.Println(name) //指定变量名,以及变量类型,未指定值的时候默认是类型零值 var age ...
- 查看Linux机器名
uname -a hostname vim /etc/sysconfig/network
- Tomcat启动错误一例org.apache.catalina.core.StandardContext resources Start Error starting static Resources
org.apache.catalina.core.StandardContext resources Start Error starting static Resources 引发原因:Eclips ...