kubernetes实战(二十七)：CentOS 8 二进制 高可用 安装 k8s 1.16.x
1. 基本说明
本文章将演示CentOS 8二进制方式安装高可用k8s 1.16.x，相对于其他版本，二进制安装方式并无太大区别。CentOS 8相对于CentOS 7操作更加方便，比如一些服务的关闭，无需修改配置文件即可永久生效，CentOS 8默认安装的内核版本是4.18，所以在安装k8s的过程中也无需在进行内核升级，系统环境也可按需升级，如果下载的是最新版的CentOS 8，系统升级也可省略。
2. 基本环境配置
主机信息
192.168.1.19 k8s-master01
192.168.1.18 k8s-master02
192.168.1.20 k8s-master03
192.168.1.88 k8s-master-lb
192.168.1.21 k8s-node01
192.168.1.22 k8s-node02
系统环境
[root@k8s-master01 ~]# cat /etc/redhat-release
CentOS Linux release 8.0. (Core)
[root@k8s-master01 ~]# uname -a
Linux k8s-master01 4.18.-.el8.x86_64 # SMP Tue Jun :: UTC x86_64 x86_64 x86_64 GNU/Linux
配置所有节点hosts文件
[root@k8s-master01 ~]# cat /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
:: localhost localhost.localdomain localhost6 localhost6.localdomain6 192.168.1.19 k8s-master01
192.168.1.18 k8s-master02
192.168.1.20 k8s-master03
192.168.1.88 k8s-master-lb
192.168.1.21 k8s-node01
192.168.1.22 k8s-node02
所有节点关闭firewalld 、dnsmasq、selinux
systemctl disable --now firewalld
systemctl disable --now dnsmasq
setenforce 0
所有节点关闭swap分区
[root@k8s-master01 ~]# swapoff -a && sysctl -w vm.swappiness=
vm.swappiness =
所有节点同步时间
ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime
echo 'Asia/Shanghai' /etc/timezone
ntpdate time2.aliyun.com
Master01节点生成ssh key
[root@k8s-master01 ~]# ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa):
Created directory '/root/.ssh'.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:6uz2kI+jcMJIUQWKqRcDRbvpVxhCW3Tmqn0NKS+lT3U root@k8s-master01
The key's randomart image is:
+---[RSA ]----+
|.o++=.o |
|.+o+ + |
|oo* . . |
|. .* + . |
|..+ + = S E |
|.= o * * . |
|. * * B . |
| = Bo+ |
| .+*oo |
+----[SHA256]-----+
Master01配置免密码登录其他节点
[root@k8s-master01 ~]# for i in k8s-master01 k8s-master02 k8s-master03 k8s-node01 k8s-node02;do ssh-copy-id -i .ssh/id_rsa.pub $i;done
所有节点安装基本工具
yum install wget jq psmisc vim net-tools yum-utils device-mapper-persistent-data lvm2 git -y
Master01下载安装文件
[root@k8s-master01 ~]# git clone https://github.com/dotbalo/k8s-ha-install.git
Cloning into 'k8s-ha-install'...
remote: Enumerating objects: , done.
remote: Counting objects: % (/), done.
remote: Compressing objects: % (/), done.
remote: Total (delta ), reused (delta ), pack-reused
Receiving objects: % (/), 19.52 MiB | 4.04 MiB/s, done.
Resolving deltas: % (/), done.
切换到1.16.x分支
git checkout manual-installation-v1..x
3. 基本组件安装
配置Docker yum源
[root@k8s-master01 k8s-ha-install]# curl https://download.docker.com/linux/centos/docker-ce.repo -o /etc/yum.repos.d/docker-ce.repo
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
--:--:-- --:--:-- --:--:--
[root@k8s-master01 k8s-ha-install]# yum makecache
CentOS- - AppStream kB/s | 6.3 MB :
CentOS- - Base kB/s | 7.9 MB :
CentOS- - Extras B/s | 2.1 kB :
Docker CE Stable - x86_64 5.8 kB/s | kB :
Last metadata expiration check: :: ago on Sat Nov :: PM CST.
Metadata cache created.
所有节点安装新版containerd
[root@k8s-master01 k8s-ha-install]# wget https://download.docker.com/linux/centos/7/x86_64/edge/Packages/containerd.io-1.2.6-3.3.el7.x86_64.rpm
---- ::-- https://download.docker.com/linux/centos/7/x86_64/edge/Packages/containerd.io-1.2.6-3.3.el7.x86_64.rpm
Resolving download.docker.com (download.docker.com)... 13.225.103.32, 13.225.103.65, 13.225.103.10, ...
Connecting to download.docker.com (download.docker.com)|13.225.103.32|:... connected.
HTTP request sent, awaiting response... OK
Length: (26M) [binary/octet-stream]
Saving to: ‘containerd.io-1.2.-3.3.el7.x86_64.rpm’ containerd.io-1.2.-3.3.el7.x86_64.rpm %[===================================================================================================================] 25.86M .55MB/s in 30s -- :: ( KB/s) - ‘containerd.io-1.2.-3.3.el7.x86_64.rpm’ saved [/] [root@k8s-master01 k8s-ha-install]# yum -y install containerd.io-1.2.-3.3.el7.x86_64.rpm
Last metadata expiration check: :: ago on Sat Nov :: PM CST.
所有节点安装最新版Docker
[root@k8s-master01 k8s-ha-install]# yum install docker-ce -y
所有节点开启Docker并设置开机自启动
[root@k8s-master01 k8s-ha-install]# systemctl enable --now docker
Created symlink /etc/systemd/system/multi-user.target.wants/docker.service → /usr/lib/systemd/system/docker.service.
[root@k8s-master01 k8s-ha-install]# docker version
Client: Docker Engine - Community
Version: 19.03.
API version: 1.40
Go version: go1.12.10
Git commit: 9013bf583a
Built: Fri Oct ::
OS/Arch: linux/amd64
Experimental: false Server: Docker Engine - Community
Engine:
Version: 19.03.
API version: 1.40 (minimum version 1.12)
Go version: go1.12.10
Git commit: 9013bf583a
Built: Fri Oct ::
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.2.
GitCommit: 894b81a4b802e4eb2a91d1ce216b8817763c29fb
runc:
Version: 1.0.-rc8
GitCommit: 425e105d5a03fabd737a126ad93d62a9eeede87f
docker-init:
Version: 0.18.
GitCommit: fec3683
4. k8s组件安装
下载kubernetes 1.16.x安装包
https://dl.k8s.io/v1.16.2/kubernetes-server-linux-amd64.tar.gz
下载etcd 3.3.15安装包
[root@k8s-master01 ~]# wget https://github.com/etcd-io/etcd/releases/download/v3.3.15/etcd-v3.3.15-linux-amd64.tar.gz
解压kubernetes安装文件
[root@k8s-master01 ~]# tar -xf kubernetes-server-linux-amd64.tar.gz --strip-components= -C /usr/local/bin kubernetes/server/bin/kube{let,ctl,-apiserver,-controller-manager,-scheduler,-proxy}
解压etcd安装文件
[root@k8s-master01 ~]# tar -zxvf etcd-v3.3.15-linux-amd64.tar.gz --strip-components= -C /usr/local/bin etcd-v3.3.15-linux-amd64/etcd{,ctl}
版本查看
[root@k8s-master01 ~]# etcd --version
etcd Version: 3.3.
Git SHA: 94745a4ee
Go Version: go1.12.9
Go OS/Arch: linux/amd64
[root@k8s-master01 ~]# kubectl version
Client Version: version.Info{Major:"", Minor:"", GitVersion:"v1.16.2", GitCommit:"c97fe5036ef3df2967d086711e6c0c405941e14b", GitTreeState:"clean", BuildDate:"2019-10-15T19:18:23Z", GoVersion:"go1.12.10", Compiler:"gc", Platform:"linux/amd64"}
The connection to the server localhost: was refused - did you specify the right host or port?
将组件发送到其他节点
MasterNodes='k8s-master02 k8s-master03'
WorkNodes='k8s-node01 k8s-node02'
for NODE in $MasterNodes; do echo $NODE; scp /usr/local/bin/kube{let,ctl,-apiserver,-controller-manager,-scheduler,-proxy} $NODE:/usr/local/bin/; scp /usr/local/bin/etcd* $NODE:/usr/local/bin/; done
for NODE in $WorkNodes; do scp /usr/local/bin/kube{let,-proxy} $NODE:/usr/local/bin/ ; done
CNI安装，下载CNI组件
wget https://github.com/containernetworking/plugins/releases/download/v0.7.5/cni-plugins-amd64-v0.7.5.tgz
所有节点创建/opt/cni/bin目录
mkdir -p /opt/cni/bin
解压cni并发送至其他节点
tar -zxf cni-plugins-amd64-v0.7.5.tgz -C /opt/cni/bin
for NODE in $MasterNodes; do ssh $NODE 'mkdir -p /opt/cni/bin'; scp /opt/cni/bin/* $NODE:/opt/cni/bin/; done
for NODE in $WorkNodes; do ssh $NODE 'mkdir -p /opt/cni/bin'; scp /opt/cni/bin/* $NODE:/opt/cni/bin/; done
5. 生成证书
下载生成证书工具
wget "https://pkg.cfssl.org/R1.2/cfssl_linux-amd64" -O /usr/local/bin/cfssl
wget "https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64" -O /usr/local/bin/cfssljson
chmod +x /usr/local/bin/cfssl /usr/local/bin/cfssljson
所有Master节点创建etcd证书目录
mkdir /etc/etcd/ssl -p
Master01节点生成etcd证书
[root@k8s-master01 pki]# pwd
/root/k8s-ha-install/pki [root@k8s-master01 pki]# cfssl gencert -initca etcd-ca-csr.json | cfssljson -bare /etc/etcd/ssl/etcd-ca
// :: [INFO] generating a new CA key and certificate from CSR
// :: [INFO] generate received request
// :: [INFO] received CSR
// :: [INFO] generating key: rsa-
// :: [INFO] encoded CSR
// :: [INFO] signed certificate with serial number [root@k8s-master01 pki]# cfssl gencert \
-ca=/etc/etcd/ssl/etcd-ca.pem \
-ca-key=/etc/etcd/ssl/etcd-ca-key.pem \
-config=ca-config.json \
-hostname=127.0.0.1,k8s-master01,k8s-master02,k8s-master03,192.168.1.19,192.168.1.18,192.168.1.20 \
-profile=kubernetes \
etcd-csr.json | cfssljson -bare /etc/etcd/ssl/etcd
// :: [INFO] generate received request
// :: [INFO] received CSR
// :: [INFO] generating key: rsa- // :: [INFO] encoded CSR
// :: [INFO] signed certificate with serial number
将证书复制到其他节点
[root@k8s-master01 pki]# MasterNodes='k8s-master02 k8s-master03'
[root@k8s-master01 pki]# WorkNodes='k8s-node01 k8s-node02'
[root@k8s-master01 pki]#
[root@k8s-master01 pki]#
[root@k8s-master01 pki]#
[root@k8s-master01 pki]# for NODE in $MasterNodes; do
ssh $NODE "mkdir -p /etc/etcd/ssl"
for FILE in etcd-ca-key.pem etcd-ca.pem etcd-key.pem etcd.pem; do
scp /etc/etcd/ssl/${FILE} $NODE:/etc/etcd/ssl/${FILE}
done
done
etcd-ca-key.pem % .0KB/s : etcd-ca.pem % .4KB/s :
etcd-key.pem % .4KB/s :
etcd.pem % .5KB/s :
etcd-ca-key.pem % .3KB/s :
etcd-ca.pem % .0KB/s :
etcd-key.pem % .1KB/s :
etcd.pem % .1KB/s :
生成kubernetes证书
所有节点创建kubernetes相关目录
mkdir -p /etc/kubernetes/pki
[root@k8s-master01 pki]# cfssl gencert -initca ca-csr.json | cfssljson -bare /etc/kubernetes/pki/ca
// :: [INFO] generating a new CA key and certificate from CSR
// :: [INFO] generate received request
// :: [INFO] received CSR
// :: [INFO] generating key: rsa-
// :: [INFO] encoded CSR
// :: [INFO] signed certificate with serial number [root@k8s-master01 pki]# cfssl gencert -ca=/etc/kubernetes/pki/ca.pem -ca-key=/etc/kubernetes/pki/ca-key.pem -config=ca-config.json -hostname=10.96.0.1,192.168.1.88,127.0.0.1,kubernetes,kubernetes.default,kubernetes.default.svc,kubernetes.default.svc.cluster,kubernetes.default.svc.cluster.local,192.168.1.19,192.168.1.18,192.168.1.20 -profile=kubernetes apiserver-csr.json | cfssljson -bare /etc/kubernetes/pki/apiserver
// :: [INFO] generate received request
// :: [INFO] received CSR
// :: [INFO] generating key: rsa-
// :: [INFO] encoded CSR
// :: [INFO] signed certificate with serial number [root@k8s-master01 pki]# cfssl gencert -initca front-proxy-ca-csr.json | cfssljson -bare /etc/kubernetes/pki/front-proxy-ca
// :: [INFO] generating a new CA key and certificate from CSR
// :: [INFO] generate received request
// :: [INFO] received CSR
// :: [INFO] generating key: rsa-
// :: [INFO] encoded CSR
// :: [INFO] signed certificate with serial number
[root@k8s-master01 pki]#
[root@k8s-master01 pki]# cfssl gencert -ca=/etc/kubernetes/pki/front-proxy-ca.pem -ca-key=/etc/kubernetes/pki/front-proxy-ca-key.pem -config=ca-config.json -profile=kubernetes front-proxy-client-csr.json | cfssljson -bare /etc/kubernetes/pki/front-proxy-client
// :: [INFO] generate received request
// :: [INFO] received CSR
// :: [INFO] generating key: rsa-
// :: [INFO] encoded CSR
// :: [INFO] signed certificate with serial number [root@k8s-master01 pki]# cfssl gencert \
-ca=/etc/kubernetes/pki/ca.pem \
-ca-key=/etc/kubernetes/pki/ca-key.pem \
-config=ca-config.json \
-profile=kubernetes \
manager-csr.json | cfssljson -bare /etc/kubernetes/pki/controller-manager
// :: [INFO] generate received request
// :: [INFO] received CSR
// :: [INFO] generating key: rsa- // :: [INFO] encoded CSR
// :: [INFO] signed certificate with serial number
// :: [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for
websites. For more information see the Baseline Requirements for the Issuance and Management
of Publicly-Trusted Certificates, v.1.1., from the CA/Browser Forum (https://cabforum.org);
specifically, section 10.2. ("Information Requirements").
[root@k8s-master01 pki]#
[root@k8s-master01 pki]# kubectl config set-cluster kubernetes \
--certificate-authority=/etc/kubernetes/pki/ca.pem \
--embed-certs=true \
--server=https://192.168.1.88:8443 \
--kubeconfig=/etc/kubernetes/controller-manager.kubeconfig kubectl config set-context system:kube-controller-manager@kubernetes \
--cluster=kubernetes \
--user=system:kube-controller-manager \
--kubeconfig=/etc/kubernetes/controller-manager.kubeconfig kubectl config use-context system:kube-controller-manager@kubernetes \
--kubeconfig=/etc/kubernetes/controller-manager.kubeconfig
Cluster "kubernetes" set.
[root@k8s-master01 pki]#
[root@k8s-master01 pki]# kubectl config set-credentials system:kube-controller-manager \
--client-certificate=/etc/kubernetes/pki/controller-manager.pem \
--client-key=/etc/kubernetes/pki/controller-manager-key.pem \
--embed-certs=true \
--kubeconfig=/etc/kubernetes/controller-manager.kubeconfig
User "system:kube-controller-manager" set.
[root@k8s-master01 pki]#
[root@k8s-master01 pki]# kubectl config set-context system:kube-controller-manager@kubernetes \
--cluster=kubernetes \
--user=system:kube-controller-manager \
--kubeconfig=/etc/kubernetes/controller-manager.kubeconfig
Context "system:kube-controller-manager@kubernetes" created.
[root@k8s-master01 pki]#
[root@k8s-master01 pki]# kubectl config use-context system:kube-controller-manager@kubernetes \
--kubeconfig=/etc/kubernetes/controller-manager.kubeconfig
Switched to context "system:kube-controller-manager@kubernetes". [root@k8s-master01 pki]# cfssl gencert \
-ca=/etc/kubernetes/pki/ca.pem \
-ca-key=/etc/kubernetes/pki/ca-key.pem \
-config=ca-config.json \
-profile=kubernetes \
scheduler-csr.json | cfssljson -bare /etc/kubernetes/pki/scheduler
// :: [INFO] generate received request
// :: [INFO] received CSR
// :: [INFO] generating key: rsa-
// :: [INFO] encoded CSR
// :: [INFO] signed certificate with serial number
// :: [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for
websites. For more information see the Baseline Requirements for the Issuance and Management
of Publicly-Trusted Certificates, v.1.1., from the CA/Browser Forum (https://cabforum.org);
specifically, section 10.2. ("Information Requirements"). [root@k8s-master01 pki]# kubectl config set-cluster kubernetes \
--certificate-authority=/etc/kubernetes/pki/ca.pem \
--embed-certs=true \
--server=https://192.168.1.88:8443 \
--kubeconfig=/etc/kubernetes/scheduler.kubeconfig
Cluster "kubernetes" set.
[root@k8s-master01 pki]#
[root@k8s-master01 pki]# kubectl config set-credentials system:kube-scheduler \
--client-certificate=/etc/kubernetes/pki/scheduler.pem \
--client-key=/etc/kubernetes/pki/scheduler-key.pem \
--embed-certs=true \
--kubeconfig=/etc/kubernetes/scheduler.kubeconfig
User "system:kube-scheduler" set.
[root@k8s-master01 pki]#
[root@k8s-master01 pki]# kubectl config set-context system:kube-scheduler@kubernetes \
--cluster=kubernetes \
--user=system:kube-scheduler \
--kubeconfig=/etc/kubernetes/scheduler.kubeconfig
Context "system:kube-scheduler@kubernetes" created.
[root@k8s-master01 pki]#
[root@k8s-master01 pki]# kubectl config use-context system:kube-scheduler@kubernetes \
--kubeconfig=/etc/kubernetes/scheduler.kubeconfig
Switched to context "system:kube-scheduler@kubernetes". [root@k8s-master01 pki]# cfssl gencert \
-ca=/etc/kubernetes/pki/ca.pem \
-ca-key=/etc/kubernetes/pki/ca-key.pem \
-config=ca-config.json \
-profile=kubernetes \
admin-csr.json | cfssljson -bare /etc/kubernetes/pki/admin
// :: [INFO] generate received request
// :: [INFO] received CSR
// :: [INFO] generating key: rsa-
// :: [INFO] encoded CSR
// :: [INFO] signed certificate with serial number
// :: [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for
websites. For more information see the Baseline Requirements for the Issuance and Management
of Publicly-Trusted Certificates, v.1.1., from the CA/Browser Forum (https://cabforum.org);
specifically, section 10.2. ("Information Requirements"). [root@k8s-master01 pki]# kubectl config set-cluster kubernetes --certificate-authority=/etc/kubernetes/pki/ca.pem --embed-certs=true --server=https://192.168.1.88:8443 --kubeconfig=/etc/kubernetes/admin.kubeconfig Cluster "kubernetes" set.
[root@k8s-master01 pki]#
[root@k8s-master01 pki]# kubectl config set-credentials kubernetes-admin --client-certificate=/etc/kubernetes/pki/admin.pem --client-key=/etc/kubernetes/pki/admin-key.pem --embed-certs=true --kubeconfig=/etc/kubernetes/admin.kubeconfig
User "kubernetes-admin" set.
[root@k8s-master01 pki]#
[root@k8s-master01 pki]# kubectl config set-context kubernetes-admin@kubernetes --cluster=kubernetes --user=kubernetes-admin --kubeconfig=/etc/kubernetes/admin.kubeconfig
Context "kubernetes-admin@kubernetes" created.
[root@k8s-master01 pki]#
[root@k8s-master01 pki]# kubectl config use-context kubernetes-admin@kubernetes --kubeconfig=/etc/kubernetes/admin.kubeconfig
Switched to context "kubernetes-admin@kubernetes". [root@k8s-master01 pki]# for NODE in k8s-master01 k8s-master02 k8s-master03; do
\cp kubelet-csr.json kubelet-$NODE-csr.json;
sed -i "s/\$NODE/$NODE/g" kubelet-$NODE-csr.json;
cfssl gencert \
-ca=/etc/kubernetes/pki/ca.pem \
-ca-key=/etc/kubernetes/pki/ca-key.pem \
-config=ca-config.json \
-hostname=$NODE \
-profile=kubernetes \
kubelet-$NODE-csr.json | cfssljson -bare /etc/kubernetes/pki/kubelet-$NODE;
rm -f kubelet-$NODE-csr.json
done
// :: [INFO] generate received request
// :: [INFO] received CSR
// :: [INFO] generating key: rsa-
// :: [INFO] encoded CSR
// :: [INFO] signed certificate with serial number
// :: [INFO] generate received request
// :: [INFO] received CSR
// :: [INFO] generating key: rsa-
// :: [INFO] encoded CSR
// :: [INFO] signed certificate with serial number
// :: [INFO] generate received request
// :: [INFO] received CSR
// :: [INFO] generating key: rsa-
// :: [INFO] encoded CSR
// :: [INFO] signed certificate with serial number [root@k8s-master01 pki]# for NODE in k8s-master01 k8s-master02 k8s-master03; do
ssh $NODE "mkdir -p /etc/kubernetes/pki"
scp /etc/kubernetes/pki/ca.pem $NODE:/etc/kubernetes/pki/ca.pem
scp /etc/kubernetes/pki/kubelet-$NODE-key.pem $NODE:/etc/kubernetes/pki/kubelet-key.pem
scp /etc/kubernetes/pki/kubelet-$NODE.pem $NODE:/etc/kubernetes/pki/kubelet.pem
rm -f /etc/kubernetes/pki/kubelet-$NODE-key.pem /etc/kubernetes/pki/kubelet-$NODE.pem
done
ca.pem % .1KB/s :
kubelet-k8s-master01-key.pem % .9KB/s :
kubelet-k8s-master01.pem % .5KB/s :
ca.pem % .2KB/s :
kubelet-k8s-master02-key.pem % .8KB/s :
kubelet-k8s-master02.pem % .5KB/s :
ca.pem % .7KB/s :
kubelet-k8s-master03-key.pem % .2KB/s :
kubelet-k8s-master03.pem % .1KB/s : [root@k8s-master01 pki]# for NODE in k8s-master01 k8s-master02 k8s-master03; do
ssh $NODE "cd /etc/kubernetes/pki && \
kubectl config set-cluster kubernetes \
--certificate-authority=/etc/kubernetes/pki/ca.pem \
--embed-certs=true \
--server=https://192.168.1.88:8443 \
--kubeconfig=/etc/kubernetes/kubelet.kubeconfig && \
kubectl config set-credentials system:node:${NODE} \
--client-certificate=/etc/kubernetes/pki/kubelet.pem \
--client-key=/etc/kubernetes/pki/kubelet-key.pem \
--embed-certs=true \
--kubeconfig=/etc/kubernetes/kubelet.kubeconfig && \
kubectl config set-context system:node:${NODE}@kubernetes \
--cluster=kubernetes \
--user=system:node:${NODE} \
--kubeconfig=/etc/kubernetes/kubelet.kubeconfig && \
kubectl config use-context system:node:${NODE}@kubernetes \
--kubeconfig=/etc/kubernetes/kubelet.kubeconfig"
done Cluster "kubernetes" set.
User "system:node:k8s-master01" set.
Context "system:node:k8s-master01@kubernetes" created.
Switched to context "system:node:k8s-master01@kubernetes".
Cluster "kubernetes" set.
User "system:node:k8s-master02" set.
Context "system:node:k8s-master02@kubernetes" created.
Switched to context "system:node:k8s-master02@kubernetes".
Cluster "kubernetes" set.
User "system:node:k8s-master03" set.
Context "system:node:k8s-master03@kubernetes" created.
Switched to context "system:node:k8s-master03@kubernetes".
创建ServiceAccount Key
[root@k8s-master01 pki]# openssl genrsa -out /etc/kubernetes/pki/sa.key
Generating RSA private key, bit long modulus ( primes)
...................................................................................+++++
...............+++++
e is (0x010001)
[root@k8s-master01 pki]# openssl rsa -in /etc/kubernetes/pki/sa.key -pubout -out /etc/kubernetes/pki/sa.pub
writing RSA key
[root@k8s-master01 pki]#
for NODE in k8s-master02 k8s-master03; do
for FILE in $(ls /etc/kubernetes/pki | grep -v etcd); do
scp /etc/kubernetes/pki/${FILE} $NODE:/etc/kubernetes/pki/${FILE};
done;
for FILE in admin.kubeconfig controller-manager.kubeconfig scheduler.kubeconfig; do
scp /etc/kubernetes/${FILE} $NODE:/etc/kubernetes/${FILE};
done;
done
admin.csr % .5KB/s :
admin-key.pem % .2KB/s :
admin.pem % .6KB/s :
apiserver.csr % .9KB/s :
apiserver-key.pem % .5KB/s :
apiserver.pem % .0KB/s :
ca.csr % .6KB/s :
ca-key.pem % .5KB/s :
ca.pem % .0KB/s :
controller-manager.csr % .4KB/s :
controller-manager-key.pem % .8KB/s :
controller-manager.pem % .7KB/s :
front-proxy-ca.csr % .2KB/s :
front-proxy-ca-key.pem % .7KB/s :
front-proxy-ca.pem % .9KB/s :
front-proxy-client.csr % .5KB/s :
front-proxy-client-key.pem % .2KB/s :
front-proxy-client.pem % .8KB/s :
kubelet-k8s-master01.csr % .6KB/s :
kubelet-k8s-master02.csr % .3KB/s :
kubelet-k8s-master03.csr % .8KB/s :
kubelet-key.pem % .2KB/s :
kubelet.pem % .4KB/s :
sa.key % .2KB/s :
sa.pub % .3KB/s :
scheduler.csr % .0KB/s :
scheduler-key.pem % .8KB/s :
scheduler.pem % .2KB/s :
admin.kubeconfig % .5MB/s :
controller-manager.kubeconfig % .1MB/s :
scheduler.kubeconfig % .6KB/s :
admin.csr % .2KB/s :
admin-key.pem % .3KB/s :
admin.pem % .7KB/s :
apiserver.csr % .4KB/s :
apiserver-key.pem % .8KB/s :
apiserver.pem % .3KB/s :
ca.csr % .4KB/s :
ca-key.pem % .6KB/s :
ca.pem % .6KB/s :
controller-manager.csr % .9KB/s :
controller-manager-key.pem % .4KB/s :
controller-manager.pem % .1KB/s :
front-proxy-ca.csr % .9KB/s :
front-proxy-ca-key.pem % .7KB/s :
front-proxy-ca.pem % .3KB/s :
front-proxy-client.csr % .6KB/s :
front-proxy-client-key.pem % .7KB/s :
front-proxy-client.pem % .1KB/s :
kubelet-k8s-master01.csr % .1KB/s :
kubelet-k8s-master02.csr % .0KB/s :
kubelet-k8s-master03.csr % .0KB/s :
kubelet-key.pem % .3KB/s :
kubelet.pem % .0KB/s :
sa.key % .5KB/s :
sa.pub % .0KB/s :
scheduler.csr % .5KB/s :
scheduler-key.pem % .3KB/s :
scheduler.pem % .3KB/s :
admin.kubeconfig % .3MB/s :
controller-manager.kubeconfig % .7MB/s :
scheduler.kubeconfig % .3MB/s :
6. Kubernetes系统组件配置
etcd配置大致相同，注意修改每个Master节点的etcd配置的主机名和IP地址
# cat /etc/etcd/etcd.config.yml
name: 'k8s-master01'
data-dir: /var/lib/etcd
wal-dir: /var/lib/etcd/wal
snapshot-count:
heartbeat-interval:
election-timeout:
quota-backend-bytes:
listen-peer-urls: 'https://192.168.1.19:2380'
listen-client-urls: 'https://192.168.1.19:2379,http://127.0.0.1:2379'
max-snapshots:
max-wals:
cors:
initial-advertise-peer-urls: 'https://192.168.1.19:2380'
advertise-client-urls: 'https://192.168.1.19:2379'
discovery:
discovery-fallback: 'proxy'
discovery-proxy:
discovery-srv:
initial-cluster: 'k8s-master01=https://192.168.1.19:2380,k8s-master02=https://192.168.1.18:2380,k8s-master03=https://192.168.1.20:2380'
initial-cluster-token: 'etcd-k8s-cluster'
initial-cluster-state: 'new'
strict-reconfig-check: false
enable-v2: true
enable-pprof: true
proxy: 'off'
proxy-failure-wait:
proxy-refresh-interval:
proxy-dial-timeout:
proxy-write-timeout:
proxy-read-timeout:
client-transport-security:
ca-file: '/etc/kubernetes/pki/etcd/etcd-ca.pem'
cert-file: '/etc/kubernetes/pki/etcd/etcd.pem'
key-file: '/etc/kubernetes/pki/etcd/etcd-key.pem'
client-cert-auth: true
trusted-ca-file: '/etc/kubernetes/pki/etcd/etcd-ca.pem'
auto-tls: true
peer-transport-security:
ca-file: '/etc/kubernetes/pki/etcd/etcd-ca.pem'
cert-file: '/etc/kubernetes/pki/etcd/etcd.pem'
key-file: '/etc/kubernetes/pki/etcd/etcd-key.pem'
peer-client-cert-auth: true
trusted-ca-file: '/etc/kubernetes/pki/etcd/etcd-ca.pem'
auto-tls: true
debug: false
log-package-levels:
log-output: default
force-new-cluster: false
所有Master节点创建etcd service并启动
[root@k8s-master01 pki]# cat /usr/lib/systemd/system/etcd.service
[Unit]
Description=Etcd Service
Documentation=https://coreos.com/etcd/docs/latest/
After=network.target [Service]
Type=notify
ExecStart=/usr/local/bin/etcd --config-file=/etc/etcd/etcd.config.yml
Restart=on-failure
RestartSec=
LimitNOFILE= [Install]
WantedBy=multi-user.target
Alias=etcd3.service [root@k8s-master01 pki]# mkdir /etc/kubernetes/pki/etcd
[root@k8s-master01 pki]# ln -s /etc/etcd/ssl/* /etc/kubernetes/pki/etcd/
[root@k8s-master01 pki]# systemctl daemon-reload
[root@k8s-master01 pki]# systemctl enable --now etcd
Created symlink /etc/systemd/system/etcd3.service → /usr/lib/systemd/system/etcd.service.
Created symlink /etc/systemd/system/multi-user.target.wants/etcd.service → /usr/lib/systemd/system/etcd.service.
高可用配置
所有Master节点安装keepalived和haproxy
yum install keepalived haproxy -y
HAProxy配置
[root@k8s-master01 pki]# cat /etc/haproxy/haproxy.cfg
global
maxconn
ulimit-n
log 127.0.0.1 local0 err
stats timeout 30s defaults
log global
mode http
option httplog
timeout connect
timeout client
timeout server
timeout http-request 15s
timeout http-keep-alive 15s frontend monitor-in
bind *:
mode http
option httplog
monitor-uri /monitor listen stats
bind *:
mode http
stats enable
stats hide-version
stats uri /stats
stats refresh 30s
stats realm Haproxy\ Statistics
stats auth admin:admin frontend k8s-master
bind 0.0.0.0:
bind 127.0.0.1:
mode tcp
option tcplog
tcp-request inspect-delay 5s
default_backend k8s-master backend k8s-master
mode tcp
option tcplog
option tcp-check
balance roundrobin
default-server inter 10s downinter 5s rise fall slowstart 60s maxconn maxqueue weight
server k8s-master01 192.168.1.19: check
server k8s-master02 192.168.1.18: check
server k8s-master03 192.168.1.20: check
KeepAlived配置
! Configuration File for keepalived
global_defs {
router_id LVS_DEVEL
}
vrrp_script chk_apiserver {
script "/etc/keepalived/check_apiserver.sh"
interval
weight -
fall
rise
}
vrrp_instance VI_1 {
state MASTER
interface ens160
mcast_src_ip 192.168.1.19
virtual_router_id
priority
advert_int
authentication {
auth_type PASS
auth_pass K8SHA_KA_AUTH
}
virtual_ipaddress {
192.168.1.88
}
track_script {
chk_apiserver
} }
健康检查配置
[root@k8s-master01 keepalived]# cat /etc/keepalived/check_apiserver.sh
#!/bin/bash err=
for k in $(seq )
do
check_code=$(pgrep kube-apiserver)
if [[ $check_code == "" ]]; then
err=$(expr $err + )
sleep
continue
else
err=
break
fi
done if [[ $err != "" ]]; then
echo "systemctl stop keepalived"
/usr/bin/systemctl stop keepalived
exit
else
exit
fi
启动HAProxy和KeepAlived
[root@k8s-master01 keepalived]# systemctl enable --now haproxy
[root@k8s-master01 keepalived]# systemctl enable --now keepalived
VIP测试
[root@k8s-master01 pki]# ping 192.168.1.88
PING 192.168.1.88 (192.168.1.88) () bytes of data.
bytes from 192.168.1.88: icmp_seq= ttl= time=1.39 ms
bytes from 192.168.1.88: icmp_seq= ttl= time=2.46 ms
bytes from 192.168.1.88: icmp_seq= ttl= time=1.68 ms
bytes from 192.168.1.88: icmp_seq= ttl= time=1.08 ms
Kubernetes组件配置
所有节点创建相关目录
[root@k8s-master01 pki]# mkdir -p /etc/kubernetes/manifests/ /etc/systemd/system/kubelet.service.d /var/lib/kubelet /var/log/kubernetes
所有Master节点创建kube-apiserver service
[root@k8s-master01 pki]# cat /usr/lib/systemd/system/kube-apiserver.service
[Unit]
Description=Kubernetes API Server
Documentation=https://github.com/kubernetes/kubernetes
After=network.target [Service]
ExecStart=/usr/local/bin/kube-apiserver \
--v= \
--logtostderr=true \
--allow-privileged=true \
--bind-address=0.0.0.0 \
--secure-port= \
--insecure-port= \
--advertise-address=192.168.1.88 \
--service-cluster-ip-range=10.96.0.0/ \
--service-node-port-range=- \
--etcd-servers=https://192.168.1.19:2379,https://192.168.1.18:2379,https://192.168.1.20:2379 \
--etcd-cafile=/etc/etcd/ssl/etcd-ca.pem \
--etcd-certfile=/etc/etcd/ssl/etcd.pem \
--etcd-keyfile=/etc/etcd/ssl/etcd-key.pem \
--client-ca-file=/etc/kubernetes/pki/ca.pem \
--tls-cert-file=/etc/kubernetes/pki/apiserver.pem \
--tls-private-key-file=/etc/kubernetes/pki/apiserver-key.pem \
--kubelet-client-certificate=/etc/kubernetes/pki/apiserver.pem \
--kubelet-client-key=/etc/kubernetes/pki/apiserver-key.pem \
--service-account-key-file=/etc/kubernetes/pki/sa.pub \
--kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname \
--enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,NodeRestriction,ResourceQuota \
--authorization-mode=Node,RBAC \
--enable-bootstrap-token-auth=true \
--requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.pem \
--proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.pem \
--proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client-key.pem \
--requestheader-allowed-names=aggregator \
--requestheader-group-headers=X-Remote-Group \
--requestheader-extra-headers-prefix=X-Remote-Extra- \
--requestheader-username-headers=X-Remote-User \
--token-auth-file=/etc/kubernetes/token.csv Restart=on-failure
RestartSec=10s
LimitNOFILE= [Install]
WantedBy=multi-user.target
[root@k8s-master01 pki]# vim /etc/kubernetes/token.csv
[root@k8s-master01 pki]# cat !$
cat /etc/kubernetes/token.csv
d7d356746b508a1a478e49968fba7947,kubelet-bootstrap,,"system:kubelet-bootstrap"
所有Master节点开启kube-apiserver
[root@k8s-master01 pki]# systemctl enable --now kube-apiserver
所有Master节点配置kube-controller-manager service
[root@k8s-master01 pki]# cat /usr/lib/systemd/system/kube-controller-manager.service
[Unit]
Description=Kubernetes Controller Manager
Documentation=https://github.com/kubernetes/kubernetes
After=network.target [Service]
ExecStart=/usr/local/bin/kube-controller-manager \
--v= \
--logtostderr=true \
--address=127.0.0.1 \
--root-ca-file=/etc/kubernetes/pki/ca.pem \
--cluster-signing-cert-file=/etc/kubernetes/pki/ca.pem \
--cluster-signing-key-file=/etc/kubernetes/pki/ca-key.pem \
--service-account-private-key-file=/etc/kubernetes/pki/sa.key \
--kubeconfig=/etc/kubernetes/controller-manager.kubeconfig \
--leader-elect=true \
--use-service-account-credentials=true \
--node-monitor-grace-period=40s \
--node-monitor-period=5s \
--pod-eviction-timeout=2m0s \
--controllers=*,bootstrapsigner,tokencleaner \
--allocate-node-cidrs=true \
--cluster-cidr=10.244.0.0/ \
--requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.pem \
--node-cidr-mask-size= Restart=always
RestartSec=10s [Install]
WantedBy=multi-user.target
所有Master节点启动kube-controller-manager
[root@k8s-master01 pki]# systemctl daemon-reload [root@k8s-master01 pki]# systemctl enable --now kube-controller-manager
Created symlink /etc/systemd/system/multi-user.target.wants/kube-controller-manager.service → /usr/lib/systemd/system/kube-controller-manager.service.
所有Master节点配置kube-scheduler service
[root@k8s-master01 pki]# cat /usr/lib/systemd/system/kube-scheduler.service
[Unit]
Description=Kubernetes Scheduler
Documentation=https://github.com/kubernetes/kubernetes
After=network.target [Service]
ExecStart=/usr/local/bin/kube-scheduler \
--v= \
--logtostderr=true \
--address=127.0.0.1 \
--leader-elect=true \
--kubeconfig=/etc/kubernetes/scheduler.kubeconfig Restart=always
RestartSec=10s [Install]
WantedBy=multi-user.target [root@k8s-master01 pki]# systemctl daemon-reload [root@k8s-master01 pki]# systemctl enable --now kube-scheduler
Created symlink /etc/systemd/system/multi-user.target.wants/kube-scheduler.service → /usr/lib/systemd/system/kube-scheduler.service.
7. TLS Bootstrapping配置
在Master01创建bootstrap
kubectl config set-cluster kubernetes --certificate-authority=/etc/kubernetes/pki/ca.pem --embed-certs=true --server=https://192.168.1.88:8443 --kubeconfig=/etc/kubernetes/bootstrap-kubelet.kubeconfig
kubectl config set-credentials tls-bootstrap-token-user --token=c8ad9c.2e4d610cf3e7426e --kubeconfig=/etc/kubernetes/bootstrap-kubelet.kubeconfig
kubectl config set-context tls-bootstrap-token-user@kubernetes --cluster=kubernetes --user=tls-bootstrap-token-user --kubeconfig=/etc/kubernetes/bootstrap-kubelet.kubeconfig
kubectl config use-context tls-bootstrap-token-user@kubernetes --kubeconfig=/etc/kubernetes/bootstrap-kubelet.kubeconfig
[root@k8s-master01 bootstrap]# pwd
/root/k8s-ha-install/bootstrap [root@k8s-master01 bootstrap]# kubectl create -f bootstrap.secret.yaml
secret/bootstrap-token-c8ad9c created
clusterrolebinding.rbac.authorization.k8s.io/kubelet-bootstrap created
clusterrolebinding.rbac.authorization.k8s.io/node-autoapprove-bootstrap created
clusterrolebinding.rbac.authorization.k8s.io/node-autoapprove-certificate-rotation created
clusterrole.rbac.authorization.k8s.io/system:kube-apiserver-to-kubelet created
clusterrolebinding.rbac.authorization.k8s.io/system:kube-apiserver created
8. Node节点配置
复制证书至Node节点
[root@k8s-master01 bootstrap]# for NODE in k8s-node01 k8s-node02; do
ssh $NODE mkdir -p /etc/kubernetes/pki /etc/etcd/ssl /etc/etcd/ssl
for FILE in etcd-ca.pem etcd.pem etcd-key.pem; do
scp /etc/etcd/ssl/$FILE $NODE:/etc/etcd/ssl/
done
for FILE in pki/ca.pem pki/ca-key.pem pki/front-proxy-ca.pem bootstrap-kubelet.kubeconfig; do
scp /etc/kubernetes/$FILE $NODE:/etc/kubernetes/${FILE}
done
done etcd-ca.pem % .0KB/s :
etcd.pem % .1KB/s :
etcd-key.pem % .9KB/s :
ca.pem % .5KB/s :
ca-key.pem % .2KB/s :
front-proxy-ca.pem % .5KB/s :
bootstrap-kubelet.kubeconfig % .1KB/s :
etcd-ca.pem % .5KB/s :
etcd.pem % .2KB/s :
etcd-key.pem % .9KB/s :
ca.pem % .8KB/s :
ca-key.pem % .0KB/s :
front-proxy-ca.pem % .9KB/s :
bootstrap-kubelet.kubeconfig % .4KB/s :
所有Node节点创建相关目录
mkdir -p /var/lib/kubelet /var/log/kubernetes /etc/systemd/system/kubelet.service.d /etc/kubernetes/manifests/
所有节点配置kubelet service（Master节点不部署Pod也可无需配置）
[root@k8s-master01 bootstrap]# vim /usr/lib/systemd/system/kubelet.service
[root@k8s-master01 bootstrap]# cat !$
cat /usr/lib/systemd/system/kubelet.service
[Unit]
Description=Kubernetes Kubelet
Documentation=https://github.com/kubernetes/kubernetes
After=docker.service
Requires=docker.service [Service]
ExecStart=/usr/local/bin/kubelet Restart=always
StartLimitInterval=
RestartSec= [Install]
WantedBy=multi-user.target
[root@k8s-master01 bootstrap]# vim /etc/systemd/system/kubelet.service.d/-kubelet.conf
[root@k8s-master01 bootstrap]# cat !$
cat /etc/systemd/system/kubelet.service.d/-kubelet.conf
[Service]
Environment="KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.kubeconfig --kubeconfig=/etc/kubernetes/kubelet.kubeconfig"
Environment="KUBELET_SYSTEM_ARGS=--network-plugin=cni --cni-conf-dir=/etc/cni/net.d --cni-bin-dir=/opt/cni/bin"
Environment="KUBELET_CONFIG_ARGS=--config=/etc/kubernetes/kubelet-conf.yml"
Environment="KUBELET_EXTRA_ARGS=--node-labels=node.kubernetes.io/node='' --pod-infra-container-image=registry.cn-hangzhou.aliyuncs.com/google_containers/pause-amd64:3.1"
ExecStart=
ExecStart=/usr/local/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_SYSTEM_ARGS $KUBELET_EXTRA_ARGS
[root@k8s-master01 bootstrap]# vim /etc/kubernetes/kubelet-conf.yml
[root@k8s-master01 bootstrap]# cat !$
cat /etc/kubernetes/kubelet-conf.yml
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
address: 0.0.0.0
port:
readOnlyPort:
authentication:
anonymous:
enabled: false
webhook:
cacheTTL: 2m0s
enabled: true
x509:
clientCAFile: /etc/kubernetes/pki/ca.pem
authorization:
mode: Webhook
webhook:
cacheAuthorizedTTL: 5m0s
cacheUnauthorizedTTL: 30s
cgroupDriver: cgroupfs
cgroupsPerQOS: true
clusterDNS:
- 10.96.0.10
clusterDomain: cluster.local
containerLogMaxFiles:
containerLogMaxSize: 10Mi
contentType: application/vnd.kubernetes.protobuf
cpuCFSQuota: true
cpuManagerPolicy: none
cpuManagerReconcilePeriod: 10s
enableControllerAttachDetach: true
enableDebuggingHandlers: true
enforceNodeAllocatable:
- pods
eventBurst:
eventRecordQPS:
evictionHard:
imagefs.available: %
memory.available: 100Mi
nodefs.available: %
nodefs.inodesFree: %
evictionPressureTransitionPeriod: 5m0s
failSwapOn: true
fileCheckFrequency: 20s
hairpinMode: promiscuous-bridge
healthzBindAddress: 127.0.0.1
healthzPort:
httpCheckFrequency: 20s
imageGCHighThresholdPercent:
imageGCLowThresholdPercent:
imageMinimumGCAge: 2m0s
iptablesDropBit:
iptablesMasqueradeBit:
kubeAPIBurst:
kubeAPIQPS:
makeIPTablesUtilChains: true
maxOpenFiles:
maxPods:
nodeStatusUpdateFrequency: 10s
oomScoreAdj: -
podPidsLimit: -
registryBurst:
registryPullQPS:
resolvConf: /etc/resolv.conf
rotateCertificates: true
runtimeRequestTimeout: 2m0s
serializeImagePulls: true
staticPodPath: /etc/kubernetes/manifests
streamingConnectionIdleTimeout: 4h0m0s
syncFrequency: 1m0s
volumeStatsAggPeriod: 1m0s
启动所有节点kubelet
systemctl daemon-reload
systemctl enable --now kubelet
查看集群状态
[root@k8s-master01 bootstrap]# kubectl get node
NAME STATUS ROLES AGE VERSION
k8s-master01 NotReady <none 13m v1.16.2
k8s-master02 NotReady <none 11m v1.16.2
k8s-master03 NotReady <none 10m v1.16.2
k8s-node01 NotReady <none 9m16s v1.16.2
k8s-node02 NotReady <none 53s v1.16.2
Kube-Proxy配置
kubectl -n kube-system create serviceaccount kube-proxy
kubectl create clusterrolebinding system:kube-proxy --clusterrole system:node-proxier --serviceaccount kube-system:kube-proxy
SECRET=$(kubectl -n kube-system get sa/kube-proxy \
--output=jsonpath='{.secrets[0].name}')
JWT_TOKEN=$(kubectl -n kube-system get secret/$SECRET \
--output=jsonpath='{.data.token}' | base64 -d)
PKI_DIR=/etc/kubernetes/pki
K8S_DIR=/etc/kubernetes
kubectl config set-cluster kubernetes --certificate-authority=/etc/kubernetes/pki/ca.pem --embed-certs=true --server=https://192.168.1.88:8443 --kubeconfig=${K8S_DIR}/kube-proxy.kubeconfig
kubectl config set-credentials kubernetes --token=${JWT_TOKEN} --kubeconfig=/etc/kubernetes/kube-proxy.kubeconfig
kubectl config set-context kubernetes --cluster=kubernetes --user=kubernetes --kubeconfig=/etc/kubernetes/kube-proxy.kubeconfig
kubectl config use-context kubernetes --kubeconfig=/etc/kubernetes/kube-proxy.kubeconfig
赋值Service文件
[root@k8s-master01 k8s-ha-install]# for NODE in k8s-master01 k8s-master02 k8s-master03; do
scp ${K8S_DIR}/kube-proxy.kubeconfig $NODE:/etc/kubernetes/kube-proxy.kubeconfig
scp kube-proxy/kube-proxy.conf $NODE:/etc/kubernetes/kube-proxy.conf
scp kube-proxy/kube-proxy.service $NODE:/usr/lib/systemd/system/kube-proxy.service
done kube-proxy.kubeconfig % .8KB/s :
kube-proxy.conf % .5KB/s :
kube-proxy.service % .5KB/s :
kube-proxy.kubeconfig % .5KB/s :
kube-proxy.conf % .1KB/s :
kube-proxy.service % .4KB/s :
kube-proxy.kubeconfig % .1KB/s :
kube-proxy.conf % .1KB/s :
kube-proxy.service % .6KB/s :
[root@k8s-master01 k8s-ha-install]#
[root@k8s-master01 k8s-ha-install]# for NODE in k8s-node01 k8s-node02; do
scp /etc/kubernetes/kube-proxy.kubeconfig $NODE:/etc/kubernetes/kube-proxy.kubeconfig
scp kube-proxy/kube-proxy.conf $NODE:/etc/kubernetes/kube-proxy.conf
scp kube-proxy/kube-proxy.service $NODE:/usr/lib/systemd/system/kube-proxy.service
done
kube-proxy.kubeconfig % .6KB/s :
kube-proxy.conf % .6KB/s :
kube-proxy.service % .5KB/s :
kube-proxy.kubeconfig % .3KB/s :
kube-proxy.conf % .2KB/s :
kube-proxy.service % .9KB/s :
所有节点启动kube-proxy
[root@k8s-master01 k8s-ha-install]# systemctl daemon-reload
[root@k8s-master01 k8s-ha-install]# systemctl enable --now kube-proxy
Created symlink /etc/systemd/system/multi-user.target.wants/kube-proxy.service → /usr/lib/systemd/system/kube-proxy.service.
9. 安装calico
[root@k8s-master01 Calico]# cd /root/k8s-ha-install/Calico/
[root@k8s-master01 Calico]# POD_CIDR="10.244.0.0/16" \
sed -i -e "s?192.168.0.0/16?$POD_CIDR?g" calico.yaml
[root@k8s-master01 Calico]# kubectl create -f calico.yaml
configmap/calico-config created
customresourcedefinition.apiextensions.k8s.io/felixconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamblocks.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/blockaffinities.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamhandles.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamconfigs.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/bgppeers.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/bgpconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ippools.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/hostendpoints.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/clusterinformations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/globalnetworkpolicies.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/globalnetworksets.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/networkpolicies.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/networksets.crd.projectcalico.org created
clusterrole.rbac.authorization.k8s.io/calico-kube-controllers created
clusterrolebinding.rbac.authorization.k8s.io/calico-kube-controllers created
clusterrole.rbac.authorization.k8s.io/calico-node created
clusterrolebinding.rbac.authorization.k8s.io/calico-node created
daemonset.apps/calico-node created
serviceaccount/calico-node created
deployment.apps/calico-kube-controllers created
serviceaccount/calico-kube-controllers created
查看Calico状态
[root@k8s-master01 Calico]# kubectl get po -n kube-system
NAME READY STATUS RESTARTS AGE
calico-kube-controllers-6d85fdfbd8-2jtj9 / Running 3m38s
calico-node-5t9kj / Running 3m38s
calico-node-9ftns / Running 3m38s
calico-node-b6rsl / Running 3m38s
calico-node-hfqrd / Running 3m38s
calico-node-lpcmp / Running 3m38s
查看集群状态
[root@k8s-master01 Calico]# kubectl cluster-info
Kubernetes master is running at https://192.168.1.88:8443 To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
[root@k8s-master01 Calico]# kubectl get node
NAME STATUS ROLES AGE VERSION
k8s-master01 Ready <none 12h v1.16.2
k8s-master02 Ready <none 13h v1.16.2
k8s-master03 Ready <none 12h v1.16.2
k8s-node01 Ready <none 12h v1.16.2
k8s-node02 Ready <none 11h v1.16.2
至此集群安装完毕，其他组件可以参考本博客其他版本安装的文档。
参考《再也不踩坑的Kubernetes实战指南》
赞助作者一杯奶茶:
Running
kubernetes实战(二十七)：CentOS 8 二进制 高可用 安装 k8s 1.16.x的更多相关文章
- Kubernetes全栈架构师（二进制高可用安装k8s集群扩展篇）--学习笔记
目录 二进制Metrics&Dashboard安装 二进制高可用集群可用性验证 生产环境k8s集群关键性配置 Bootstrapping: Kubelet启动过程 Bootstrapping: ...
- Kubernetes全栈架构师（二进制高可用安装k8s集群部署篇）--学习笔记
目录 二进制高可用基本配置 二进制系统和内核升级 二进制基本组件安装 二进制生成证书详解 二进制高可用及etcd配置 二进制K8s组件配置 二进制使用Bootstrapping自动颁发证书 二进制No ...
- kubernetes实战(三十)：CentOS 8 二进制 高可用 安装 k8s 1.17.x
1. 基本说明 本文章将演示CentOS 8二进制方式安装高可用k8s 1.17.x,相对于其他版本,二进制安装方式并无太大区别. 2. 基本环境配置 主机信息 192.168.1.19 k8s-ma ...
- Kubernetes全栈架构师（Kubeadm高可用安装k8s集群）--学习笔记
目录 k8s高可用架构解析 Kubeadm基本环境配置 Kubeadm系统及内核升级 Kubeadm基本组件安装 Kubeadm高可用组件安装 Kubeadm集群初始化 高可用Master及Token ...
- kubernetes教程第一章-kubeadm高可用安装k8s集群
目录 Kubeadm高可用安装k8s集群 kubeadm高可用安装1.18基本说明 k8s高可用架构解析 kubeadm基本环境配置 kubeadm基本组件安装 kubeadm集群初始化 高可用Mas ...
- Kubernetes实战指南（三十四）： 高可用安装K8s集群1.20.x
@ 目录 1. 安装说明 2. 节点规划 3. 基本配置 4. 内核配置 5. 基本组件安装 6. 高可用组件安装 7. 集群初始化 8. 高可用Master 9. 添加Node节点 10. Cali ...
- 【Containerd版】Kubeadm高可用安装K8s集群1.23+
目录 基本环境配置 节点规划 网段规划及软件版本 基本配置 内核升级配置 K8s组件及Runtime安装 Containerd安装 K8s组件安装 高可用实现 集群初始化 Master01初始化 添加 ...
- kubernetes实战(二十八)：Kubernetes一键式资源管理平台Ratel安装及使用
1. Ratel是什么? Ratel是一个Kubernetes资源平台,基于管理Kubernetes的资源开发,可以管理Kubernetes的Deployment.DaemonSet.Stateful ...
- Kubernetes实战(一)：k8s v1.11.x v1.12.x 高可用安装
说明:部署的过程中请保证每个命令都有在相应的节点执行,并且执行成功,此文档已经帮助几十人(仅包含和我取得联系的)快速部署k8s高可用集群,文档不足之处也已更改,在部署过程中遇到问题请先检查是否遗忘某个 ...
随机推荐
- 编程范式 --- 面向协议编程（Protocol Oriented Programming，简称POP）
面向协议编程(Protocol Oriented Programming,简称POP) 是Swift的一种编程范式,Apple于2015年WWDC踢出 在Swift的标准库中,能见到大量POP的影子 ...
- 利用百度云接口实现车牌识别·python
一个小需求---实现车牌识别. 目前有两个想法 1. 调云在线的接口或者使用SDK做开发(配置环境和变异第三方库麻烦,当然使用python可以避免这些问题) 2. 自己实现车牌识别算法(复杂) 一开始 ...
- .NET开发者必须学习.NET Core
很多的.NET开发者在接触.Net Core之前,对于linux系统一点也不了解,也未曾有过主动去学习的念头.在接触了.Net Core之后才会慢慢学习linux相关知识,很多同学想转Java,这个很 ...
- TensorFlow基本计算单元与基本操作
在学习深度学习等知识之前,首先得了解著名的框架TensorFlow里面的一些基础知识,下面首先看一下这个框架的一些基本用法. import tensorflow as tf a = 3 # Pytho ...
- svg foreignObject的作用（文本换行，生成图片）
SVG内部利用foreignObject嵌入XHTML元素 <foreignObject>元素的作用是可以在其中使用具有其它XML命名空间的XML元素,换句话说借助<foreignO ...
- Redis开发与运维：linux安装
Linux 安装 我的系统是inux 系统,官网下载 https://redis.io/download redis-5.0.5.tar.gz 解压: 编译安装: 官网和文档说得已经很清楚了,现在就执 ...
- 阿里云Centos操作Mysql
1. CentOS7 yum方式安装MySQL5.7 2.Centos下mysql导入sql 3.Centos设置远程 4.CentOS下mysql数据库常用命令总结 5.centos出现“Fire ...
- 如何选择一款适合自己操作系统、Windows、Mac还是Linux？
如何选择一款适合自己操作系统.Windows.Mac还是Linux? 作者:我们都很努力着 简介:电脑已经逐渐离不开我们生活,但是如何选择一个我们最佳,最适合的电脑操作系统就成了一些困难选择人士的一个 ...
- A-08 拉格朗日对偶性
目录 拉格朗日对偶性 一.原始问题 1.1 约束最优化问题 1.2 广义拉格朗日函数 1.3 约束条件的考虑 二.对偶问题 三.原始问题和对偶问题的关系 3.1 定理1 3.2 推论1 3.3 定理2 ...
- markdown公式转为知乎格式
在知乎中写技术类文章,经常会用到markdown知乎文章可以导入markdown格式,但是不支持Latex公式.知乎大神提供了替代方案: https://zhuanlan.zhihu.com/ ...