kubernetes1.7.6 ha高可用部署

写在前面：

1. 该文章部署方式为二进制部署。

2. 版本信息 k8s 1.7.6，etcd 3.2.9

3. 高可用部分 etcd做高可用集群、kube-apiserver 为无状态服务使用haproxy做负载均衡，kube-controller-manager和kube-scheduler使用自身的选举功能，无需考虑高可用问题。

环境说明：

本环境中网络说明，宿主机及node网段为192.168.1.x/24,service cluster网段为172.16.x.x/16 ,pod网段为172.17.x.x/16,下面会用到。

主机名	ip地址	服务	备注
master1	192.168.1.18	etcd flanneld kube-apiserver kube-controller-manager kube-scheduler haproxy keepalived	VIP 192.168.1.24作为apiserver的浮动ip
master2	192.168.1.19	etcd flanneld kube-apiserver kube-controller-manager kube-scheduler haproxy keepalived	VIP 192.168.1.24作为apiserver的浮动ip
master3	192.168.1.20	etcd flanneld kube-apiserver kube-controller-manager kube-scheduler
node1	192.168.1.21	flanneld docker kube-proxy kubelet harbor
node2	192.168.1.22	flanneld docker kube-proxy kubelet harbor
node3	192.168.1.23	flanneld docker kube-proxy kubelet harbor

步骤：

1. 证书及kubeconfig文件生成(该操作在任何一台master上执行即可)

kubernetes 系统的各组件需要使用 TLS 证书对通信进行加密，本文档使用 CloudFlare 的 PKI 工具集 cfssl 来生成 Certificate Authority (CA) 和其它证书；

生成的 CA 证书和秘钥文件如下：

ca-key.pem
ca.pem
kubernetes-key.pem
kubernetes.pem
kube-proxy.pem
kube-proxy-key.pem
admin.pem
admin-key.pem

使用证书的组件如下：

etcd：使用 ca.pem、kubernetes-key.pem、kubernetes.pem；
kube-apiserver：使用 ca.pem、kubernetes-key.pem、kubernetes.pem；
kubelet：使用 ca.pem；
kube-proxy：使用 ca.pem、kube-proxy-key.pem、kube-proxy.pem；
kubectl：使用 ca.pem、admin-key.pem、admin.pem；

证书生成需要使用cfssl，下面安装cfssl：

[root@k8s-master01 ~]# wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64

[root@k8s-master01 ~]# chmod +x cfssl_linux-amd64

[root@k8s-master01 ~]# mv cfssl_linux-amd64 /usr/bin/cfssl

[root@k8s-master01 ~]# wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64

[root@k8s-master01 ~]# chmod +x cfssljson_linux-amd64

[root@k8s-master01 ~]# mv cfssljson_linux-amd64 /usr/bin/cfssljson

[root@k8s-master01 ~]# wget https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64

[root@k8s-master01 ~]# chmod +x cfssl-certinfo_linux-amd64

[root@k8s-master01 ~]# mv cfssl-certinfo_linux-amd64 /usr/bin/cfssl-certinfo

创建 CA (Certificate Authority)

创建 CA 配置文件

[root@k8s-master01 ~]# mkdir /opt/ssl

[root@k8s-master01 ~]# cd /opt/ssl

[root@k8s-master01 ~]# cfssl print-defaults config > config.json

[root@k8s-master01 ~]# cfssl print-defaults csr > csr.json

[root@k8s-master01 ~]# cat ca-config.json

{

  "signing": {

    "default": {

      "expiry": "8760h"

    },

    "profiles": {

      "kubernetes": {

        "usages": [

            "signing",

            "key encipherment",

            "server auth",

            "client auth"

        ],

        "expiry": "8760h"

      }

    }

  }

}

创建 CA 证书签名请求

[root@k8s-master01 ~]# cat ca-csr.json

{

  "CN": "kubernetes",

  "key": {

    "algo": "rsa",

    "size":

  },

  "names": [

    {

      "C": "CN",

      "ST": "BeiJing",

      "L": "BeiJing",

      "O": "k8s",

      "OU": "System"

    }

  ]

}

生成 CA 证书和私钥

[root@k8s-master01 ~]# cfssl gencert -initca ca-csr.json | cfssljson -bare ca

[root@k8s-master01 ~]# ls ca*

ca-config.json  ca.csr  ca-csr.json  ca-key.pem  ca.pem

创建 kubernetes 证书

创建 kubernetes 证书签名请求

[root@k8s-master01 ~]# cat kubernetes-csr.json

{

    "CN": "kubernetes",

    "hosts": [

      "127.0.0.1",

      "192.168.1.18",

      "192.168.1.19",

      "192.168.1.20",

      "192.168.1.21",

      "192.168.1.22",

      "192.168.1.23",

      "172.16.0.1",

      "kubernetes",

      "kubernetes.default",

      "kubernetes.default.svc",

      "kubernetes.default.svc.cluster",

      "kubernetes.default.svc.cluster.local"

    ],

    "key": {

        "algo": "rsa",

        "size":

    },

    "names": [

        {

            "C": "CN",

            "ST": "BeiJing",

            "L": "BeiJing",

            "O": "k8s",

            "OU": "System"

        }

    ]

}

如果 hosts 字段不为空则需要指定授权使用该证书的 IP 或域名列表，由于该证书后续被 etcd 集群和 kubernetes master集群使用，所以上面分别指定了 etcd 集群、kubernetes master 集群的主机 IP 和 kubernetes 服务的服务 IP（一般是kue-apiserver 指定的 service-cluster-ip-range 网段的第一个IP，如本环境中的172.16.0.1。

生成 kubernetes 证书和私钥

$ cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kubernetes-csr.json | cfssljson -bare kubernetes

$ ls kuberntes*

kubernetes.csr  kubernetes-csr.json  kubernetes-key.pem  kubernetes.pem

创建 admin 证书

创建 admin 证书签名请求

$ cat admin-csr.json

{

  "CN": "admin",

  "hosts": [],

  "key": {

    "algo": "rsa",

    "size":

  },

  "names": [

    {

      "C": "CN",

      "ST": "BeiJing",

      "L": "BeiJing",

      "O": "system:masters",

      "OU": "System"

    }

  ]

}

生成 admin 证书和私钥

$ cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes admin-csr.json | cfssljson -bare admin

$ ls admin*

admin.csr  admin-csr.json  admin-key.pem  admin.pem

创建 kube-proxy 证书

创建 kube-proxy 证书签名请求

$ cat kube-proxy-csr.json

{

  "CN": "system:kube-proxy",

  "hosts": [],

  "key": {

    "algo": "rsa",

    "size":

  },

  "names": [

    {

      "C": "CN",

      "ST": "BeiJing",

      "L": "BeiJing",

      "O": "k8s",

      "OU": "System"

    }

  ]

}

生成 kube-proxy 客户端证书和私钥

$ cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes  kube-proxy-csr.json | cfssljson -bare kube-proxy

$ ls kube-proxy*

kube-proxy.csr  kube-proxy-csr.json  kube-proxy-key.pem  kube-proxy.pem

分发证书

将生成的证书和秘钥文件（后缀名为.pem）拷贝到所有机器的 /etc/kubernetes/ssl 目录下备用；

[root@k8s-master01 ~]# cd /opt/ssl/

[root@k8s-master01 ssl]# mkdir -p /etc/kubernetes/ssl/[root@k8s-master01 ssl]# cp * /etc/kubernetes/ssl/

[root@k8s-master01 ssl]# for i in `seq  `; do  scp -r /etc/kubernetes/ 192.168..$i:/etc/;done

创建 kubeconfig 文件

配置kubectl的kubeconfig文件

文件会生产在 /root/.kube/config

#配置 kubernetes 集群

[root@k8s-master01 ~]# kubectl config set-cluster kubernetes \

>   --certificate-authority=/etc/kubernetes/ssl/ca.pem \

>   --embed-certs=true \

>   --server=https://192.168.1.24:6444

Cluster "kubernetes" set.

#配置 客户端认证

[root@k8s-master01 ~]# kubectl config set-credentials admin \

>   --client-certificate=/etc/kubernetes/ssl/admin.pem \

>   --embed-certs=true \

>   --client-key=/etc/kubernetes/ssl/admin-key.pem

User "admin" set.

[root@k8s-master01 ~]# kubectl config set-context kubernetes \

>   --cluster=kubernetes \

>   --user=admin

Context "kubernetes" created.

[root@k8s-master01 ~]# kubectl config use-context kubernetes

Switched to context "kubernetes".

#分发文件

[root@k8s-master01 ~]# for i in `seq  `;do scp -r /root/.kube 192.168..$i:/root/;done

config                                                                                              %      .1KB/s   :

config                                                                                              %      .1KB/s   :

config                                                                                              %      .1KB/s   :

config                                                                                              %      .1KB/s   :

config                                                                                              %      .1KB/s   :

[root@k8s-master01 ~]#

kubelet、kube-proxy 等 Node 机器上的进程与 Master 机器的 kube-apiserver 进程通信时需要认证和授权；

kubernetes 1.4 开始支持由 kube-apiserver 为客户端生成 TLS 证书的 TLS Bootstrapping 功能，这样就不需要为每个客户端生成证书了；该功能当前仅支持为 kubelet 生成证书；

创建 TLS Bootstrapping Token

Token auth file

Token可以是任意的包涵128 bit的字符串，可以使用安全的随机数发生器生成。

[root@k8s-master01 ssl]# cd /etc/kubernetes/

[root@k8s-master01 kubernetes]# export BOOTSTRAP_TOKEN=$(head -c  /dev/urandom | od -An -t x | tr -d ' ')

[root@k8s-master01 kubernetes]# cat > token.csv <<EOF

> ${BOOTSTRAP_TOKEN},kubelet-bootstrap,,"system:kubelet-bootstrap"

> EOF

[root@k8s-master01 kubernetes]# ls

ssl  token.csv

[root@k8s-master01 kubernetes]# cat token.csv

bd962dfaa4b87d896c4e944f113428d3,kubelet-bootstrap,,"system:kubelet-bootstrap"

[root@k8s-master01 kubernetes]#

将token.csv发到所有机器（Master 和 Node）的 /etc/kubernetes/ 目录。

[root@k8s-master01 kubernetes]# for i in `seq  `; do scp token.csv 192.168..$i:/etc/kubernetes/;done

token.csv                                                                                           %        .1KB/s   :

token.csv                                                                                           %        .1KB/s   :

token.csv                                                                                           %        .1KB/s   :

token.csv                                                                                           %        .1KB/s   :

token.csv                                                                                           %        .1KB/s   :

[root@k8s-master01 kubernetes]#

创建 kubelet bootstrapping kubeconfig 文件

kubelet 启动时向 kube-apiserver 发送 TLS bootstrapping 请求，需要先将 bootstrap token 文件中的 kubelet-bootstrap 用户赋予 system:node-bootstrapper 角色，然后 kubelet 才有权限创建认证请求(certificatesigningrequests)。

先创建认证请求 user 为 master 中 token.csv 文件里配置的用户只需在一个node中创建一次就可以
Master节点执行

[root@k8s-master01 bin]# kubectl create clusterrolebinding kubelet-bootstrap --clusterrole=system:node-bootstrapper --user=kubelet-bootstrap

创建kubelet kubeconfig文件

拷贝kubectl二进制文件
[root@k8s-master01 bin]# cp kubectl /usr/bin/

[root@k8s-master01 bin]# cd /etc/kubernetes/

#配置集群

[root@k8s-master01 kubernetes]# kubectl config set-cluster kubernetes \

>   --certificate-authority=/etc/kubernetes/ssl/ca.pem \

>   --embed-certs=true \

>   --server=https://192.168.1.24:6444 \

>   --kubeconfig=bootstrap.kubeconfig

Cluster "kubernetes" set.

#配置客户端认证

[root@k8s-master01 kubernetes]# kubectl config set-credentials kubelet-bootstrap \

>   --token=bd962dfaa4b87d896c4e944f113428d3 \

>   --kubeconfig=bootstrap.kubeconfig

User "kubelet-bootstrap" set.

#配置关联

[root@k8s-master01 kubernetes]# kubectl config set-context default \

>   --cluster=kubernetes \

>   --user=kubelet-bootstrap \

>   --kubeconfig=bootstrap.kubeconfig

Context "default" created.

#配置默认关联

[root@k8s-master01 kubernetes]# kubectl config use-context default --kubeconfig=bootstrap.kubeconfig

Switched to context "default".

[root@k8s-master01 kubernetes]# ls

bootstrap.kubeconfig  ssl  token.csv

#分发文件

[root@k8s-master01 kubernetes]# for i in `seq  `; do scp bootstrap.kubeconfig 192.168..$i:/etc/kubernetes/;done

bootstrap.kubeconfig                                                                                %      .1KB/s   :

bootstrap.kubeconfig                                                                                %      .1KB/s   :

bootstrap.kubeconfig                                                                                %      .1KB/s   :

bootstrap.kubeconfig                                                                                %      .1KB/s   :

bootstrap.kubeconfig                                                                                %      .1KB/s   :

[root@k8s-master01 kubernetes]#

创建kube-proxy kubeconfig文件

[root@k8s-master01 ~]# cd /etc/kubernetes/

#Node节点 配置集群

[root@k8s-master01 kubernetes]# kubectl config set-cluster kubernetes \

>   --certificate-authority=/etc/kubernetes/ssl/ca.pem \

>   --embed-certs=true \

>   --server=https://192.168.1.24:6444 \

>   --kubeconfig=kube-proxy.kubeconfig

Cluster "kubernetes" set.

#配置客户端认证

[root@k8s-master01 kubernetes]# kubectl config set-credentials kube-proxy \

>   --client-certificate=/etc/kubernetes/ssl/kube-proxy.pem \

>   --client-key=/etc/kubernetes/ssl/kube-proxy-key.pem \

>   --embed-certs=true \

>   --kubeconfig=kube-proxy.kubeconfig

User "kube-proxy" set.

#配置关联

[root@k8s-master01 kubernetes]# kubectl config set-context default \

>   --cluster=kubernetes \

>   --user=kube-proxy \

>   --kubeconfig=kube-proxy.kubeconfig

Context "default" created.

#配置默认关联

[root@k8s-master01 kubernetes]# kubectl config use-context default --kubeconfig=kube-proxy.kubeconfig

Switched to context "default".

[root@k8s-master01 kubernetes]# ls

bootstrap.kubeconfig  kube-proxy.kubeconfig  ssl  token.csv

[root@k8s-master01 kubernetes]# 

#分发文件到所有node节点即可
[root@k8s-master01 kubernetes]# for i in `seq 19 23`; do scp kube-proxy.kubeconfig 192.168.1.$i:/etc/kubernetes/;done

　kube-proxy.kubeconfig 100% 6272 6.1KB/s 00:00
　kube-proxy.kubeconfig 100% 6272 6.1KB/s 00:00
kube-proxy.kubeconfig 100% 6272 6.1KB/s 00:00
kube-proxy.kubeconfig 100% 6272 6.1KB/s 00:00
kube-proxy.kubeconfig 100% 6272 6.1KB/s 00:00

2. etcd高可用部署

3. master节点配置

安装

[root@k8s-master01 src]# tar zxvf flannel-v0.9.0-linux-amd64.tar.gz

flanneld

mk-docker-opts.sh

README.md

[root@k8s-master01 src]# mv flanneld /usr/bin/

[root@k8s-master01 src]# mv mk-docker-opts.sh /usr/bin/

[root@k8s-master01 src]# for i in `seq  `;do scp /usr/bin/flanneld /usr/bin/mk-docker-opts.sh 192.168..$i:/usr/bin/ ;done

flanneld                                                                                            %   33MB  .9MB/s   :

mk-docker-opts.sh                                                                                   %      .1KB/s   :

flanneld                                                                                            %   33MB  .9MB/s   :

mk-docker-opts.sh                                                                                   %      .1KB/s   :

flanneld                                                                                            %   33MB  .9MB/s   :

mk-docker-opts.sh                                                                                   %      .1KB/s   :

flanneld                                                                                            %   33MB  .9MB/s   :

mk-docker-opts.sh                                                                                   %      .1KB/s   :

flanneld                                                                                            %   33MB  .9MB/s   :

mk-docker-opts.sh                                                                                   %      .1KB/s   :

[root@k8s-master01 src]#

所有master节点分发二进制程序

[root@k8s-master01 bin]# for i in `seq  `;do scp kube-apiserver kube-controller-manager kube-scheduler 192.168..$i:/usr/bin/;donekube-apiserver                                                                                      %  176MB  .2MB/s   :

kube-controller-manager                                                                             %  131MB  .3MB/s   :

kube-scheduler                                                                                      %   73MB  .6MB/s   :

kube-apiserver                                                                                      %  176MB  .8MB/s   :

kube-controller-manager                                                                             %  131MB  .3MB/s   :

kube-scheduler                                                                                      %   73MB  .6MB/s   :

kube-apiserver                                                                                      %  176MB  .8MB/s   :

kube-controller-manager                                                                             %  131MB  .3MB/s   :

kube-scheduler                                                                                      %   73MB  .6MB/s   :

添加CA证书到系统信任库

使用动态CA配置

update-ca-trust force-enable

拷贝ca根证书到指定目录

cp /etc/kubernetes/ssl/ca.pem /etc/pki/ca-trust/source/anchors/

生效

update-ca-trust extract

5.3 配置flannel的ip段

etcd节点执行此网段为上面提到的pod网段

[root@k8s-master01 src]# etcdctl --endpoint https://192.168.1.18:2379 set /flannel/network/config '{"Network":"172.17.0.0/16"}'

{"Network":"172.17.0.0/16"}

[root@k8s-master01 src]#

5.4 配置flannel

设置flanneld.service

[root@k8s-master01 system]# cat /usr/lib/systemd/system/flanneld.service

[Unit]

Description=Flanneld overlay address etcd agent

After=network.target

After=network-online.target

Wants=network-online.target

After=etcd.service

Before=docker.service

[Service]

Type=notify

EnvironmentFile=/etc/sysconfig/flanneld

EnvironmentFile=-/etc/sysconfig/docker-network

ExecStart=/usr/bin/flanneld-start $FLANNEL_OPTIONS

ExecStartPost=/usr/bin/mk-docker-opts.sh -k DOCKER_NETWORK_OPTIONS -d /run/flannel/docker

Restart=on-failure

[Install]

WantedBy=multi-user.target

RequiredBy=docker.service

[root@k8s-master01 system]#

配置文件：

[root@k8s-master01 system]# cat  /etc/sysconfig/flanneld

FLANNEL_ETCD_ENDPOINTS="https://192.168.1.18:2379,https://192.168.1.19:2379,https://192.168.1.20:2379"

FLANNEL_ETCD_PREFIX="/flannel/network"

FLANNEL_OPTIONS="--iface=eth0"

#iface为物理网卡名

[root@k8s-master01 system]# cat /etc/sysconfig/docker-network
CKER_NETWORK_OPTIONS=

#可以为空

[root@k8s-master01 system]# cat /usr/bin/flanneld-start
#!/bin/sh

exec /usr/bin/flanneld \
-etcd-endpoints=${FLANNEL_ETCD_ENDPOINTS:-${FLANNEL_ETCD}} \
-etcd-prefix=${FLANNEL_ETCD_PREFIX:-${FLANNEL_ETCD_KEY}} \
"$@"

[root@k8s-master01 system]# chmod +x /usr/bin/flanneld-start

确保docker已停止

systemctl stop docker

启动flanneld服务

systemctl daemon-reload

systemctl enable flanneld

systemctl start flanneld

验证

[root@k8s-master01 system]# ifconfig flannel0

flannel0: flags=<UP,POINTOPOINT,RUNNING,NOARP,MULTICAST>  mtu

        inet 172.17.2.0  netmask 255.255.0.0  destination 172.17.2.0

        unspec ---------------  txqueuelen   (UNSPEC)

        RX packets   bytes  (0.0 B)

        RX errors   dropped   overruns   frame

        TX packets   bytes  (0.0 B)

        TX errors   dropped  overruns   carrier   collisions

配置kube-apiserver

创建日志目录

[root@k8s-master01 ~]# mkdir /var/log/kubernetes

配置service文件

[root@k8s-master01 system]# cat /usr/lib/systemd/system/kube-apiserver.service

[Unit]

Description=Kubernetes API Server

Documentation=https://github.com/GoogleCloudPlatform/kubernetes

After=network.target

[Service]

User=root

ExecStart=/usr/bin/kube-apiserver \

  --admission-control=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,ResourceQuota \

  --advertise-address=192.168.1.18 \

  --allow-privileged=true \

  --apiserver-count= \

  --audit-log-maxage= \

  --audit-log-maxbackup= \

  --audit-log-maxsize= \

  --audit-log-path=/var/lib/audit.log \

  --authorization-mode=RBAC \

  --bind-address=192.168.1.18 \

  --client-ca-file=/etc/kubernetes/ssl/ca.pem \

  --enable-swagger-ui=true \

  --etcd-cafile=/etc/kubernetes/ssl/ca.pem \

  --etcd-certfile=/etc/kubernetes/ssl/kubernetes.pem \

  --etcd-keyfile=/etc/kubernetes/ssl/kubernetes-key.pem \

  --etcd-servers=https://192.168.1.18:2379,https://192.168.1.19:2379,https://192.168.1.20:2379 \

  --event-ttl=1h \

  --kubelet-https=true \

  --insecure-bind-address=192.168.1.18 \

  --runtime-config=rbac.authorization.k8s.io/v1alpha1 \

  --service-account-key-file=/etc/kubernetes/ssl/ca-key.pem \

  --service-cluster-ip-range=172.16.0.0/ \

  --service-node-port-range=- \

  --tls-cert-file=/etc/kubernetes/ssl/kubernetes.pem \

  --tls-private-key-file=/etc/kubernetes/ssl/kubernetes-key.pem \

  --experimental-bootstrap-token-auth \

  --token-auth-file=/etc/kubernetes/token.csv \

  --logtostderr=false \

  --log-dir=/var/log/kubernetes \

  --v=

Restart=on-failure

RestartSec=

Type=notify

LimitNOFILE=

[Install]

WantedBy=multi-user.target

[root@k8s-master01 system]# 
#其中--service-cluster-ip-range=172.16.0.0/16 就是上面提到的service 网段，这里面要注意的是 --service-node-port-range=30000-32000
这个地方是 映射外部端口时 的端口范围，随机映射也在这个范围内映射，指定映射端口必须也在这个范围内。

#启动服务

systemctl daemon-reload

systemctl enable kube-apiserver

systemctl start kube-apiserver

systemctl status kube-apiserver

配置kube-controller-manager

配置service文件

[root@k8s-master01 kubernetes]# cat  /usr/lib/systemd/system/kube-controller-manager.service

[Unit]

Description=Kubernetes Controller Manager

Documentation=https://github.com/GoogleCloudPlatform/kubernetes

[Service]

ExecStart=/usr/bin/kube-controller-manager \

  --address=0.0.0.0 \

  --master=http://192.168.1.24:8081 \

  --allocate-node-cidrs=true \

  --service-cluster-ip-range=172.16.0.0/ \

  --cluster-cidr=172.17.0.0/ \

  --cluster-name=kubernetes \

  --cluster-signing-cert-file=/etc/kubernetes/ssl/ca.pem \

  --cluster-signing-key-file=/etc/kubernetes/ssl/ca-key.pem \

  --service-account-private-key-file=/etc/kubernetes/ssl/ca-key.pem \

  --root-ca-file=/etc/kubernetes/ssl/ca.pem \

  --leader-elect=true \

  --logtostderr=false \

  --log-dir=/var/log/kubernetes \

  --v=

Restart=on-failure

RestartSec=

[Install]

WantedBy=multi-user.target

[root@k8s-master01 kubernetes]#

#启动服务

systemctl daemon-reload

systemctl enable kube-controller-manager

systemctl start kube-controller-manager

systemctl status kube-controller-manager

配置kube-controller-scheduler

4. node节点配置

5. 安装私有仓库harbor并高可用配置

6. 安装dns插件

7. 安装 dashboard

8. 安装监控插件